Image Based Rendering
description
Transcript of Image Based Rendering
![Page 1: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/1.jpg)
Image Based Renderingan overview
![Page 2: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/2.jpg)
2
Photographs
• We have tools that acquire and tools that display photographs at a convincing quality level
![Page 3: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/3.jpg)
3
![Page 4: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/4.jpg)
4
![Page 5: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/5.jpg)
5
![Page 6: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/6.jpg)
6
![Page 7: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/7.jpg)
7
![Page 8: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/8.jpg)
8
![Page 9: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/9.jpg)
9
![Page 10: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/10.jpg)
10
![Page 11: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/11.jpg)
11
Photographs
• We have tools that acquire and tools that display photographs at a convincing quality level, for almost 100 years now
![Page 12: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/12.jpg)
12
Sergei Mikhailovich Prokudin-Gorskii.
A Settler's Family, ca. 1907-1915.
![Page 13: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/13.jpg)
13
Sergei Mikhailovich Prokudin-Gorskii.Tea Factory in Chakva.
Chinese Foreman Lau-Dzhen-Dzhau.ca. 1907-1915.
![Page 14: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/14.jpg)
14
Sergei Mikhailovich Prokudin-Gorskii.
The Emir of Bukhara, 1911.
![Page 15: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/15.jpg)
15
RGB in early 1900’s
![Page 16: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/16.jpg)
16
Plenoptic function
• Defines all the rays– through any point in space (x, y, z)– with any orientation (θ, φ)– over all wavelenghts (λ)– at any given moment in time (t)
),,,,,,( tzyxP
![Page 17: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/17.jpg)
17
IBR summary
explicitimplicit
geom
etric
mod
el
text
ure
map
ping
pano
ram
as
view
mor
phin
g
3D im
age
war
ping
ray
data
base
s
Representation of plenoptic function
![Page 18: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/18.jpg)
18
Lightfield – Lumigraph approach[Levoy96, Gortler96]
• Take all photographs you will ever need to display
• Model becomes database of rays• Rendering becomes database querying
![Page 19: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/19.jpg)
19
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 20: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/20.jpg)
20
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 21: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/21.jpg)
21
From 7D to 4D
• Static scene, t constant• λ approximated with RGB• consider only convex hull of objects, so the
origin of the ray does not matter
),,,,,,( tzyxP
![Page 22: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/22.jpg)
22
4D Lightfield / Lumigraph
![Page 23: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/23.jpg)
23
Discreet 4D Lightfield
![Page 24: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/24.jpg)
24
Lightfield: set of images with COPs on regular grid
![Page 25: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/25.jpg)
25
or Lightfield: set of images of a point seen at various angles
![Page 26: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/26.jpg)
26
Depth correction of rays
![Page 27: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/27.jpg)
27
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 28: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/28.jpg)
28
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 29: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/29.jpg)
29
Construction from dense set of photographs
![Page 30: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/30.jpg)
30
Construction from sparse set of photographs
acquisition stage
camera positions
blue screening space carving
![Page 31: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/31.jpg)
31
Filling in gaps using pull-push algorithm
• Pull phase • low res levels are
created• gaps are shrunk
• Push phase• gaps at high res levels
are filled using low res levels
![Page 32: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/32.jpg)
32
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 33: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/33.jpg)
33
Overview
• Introduction• Lightfield – Lumigraph
– definition– construction– compression
![Page 34: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/34.jpg)
34
Compression
• Large size uncompressed: 1.125GB– 32x32 (s, t) x 256x256 (u, v) x 6 faces x 3 B
• Compression– jpeg + mpeg (200:1 to 6MB)– or vector quantization + entropy encoding
![Page 35: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/35.jpg)
35
Vector Quantization (VQ)
• Principle– codebook made of codewords– replace actual word with closest codeword
• Implementation– training on representative set of words to derive best
codebook– compression: replacing word with index to closest
codeword– decompression: retrieve indexed codeword from
codebook
![Page 36: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/36.jpg)
36
Lightfield compression using VQ
![Page 38: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/38.jpg)
38
Motivation – rendering from images
• Given– left image– right image
• Create intermediate images– simulates camera
movement[Seitz96]
![Page 39: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/39.jpg)
39
Previous work
• Panoramas ([Chen95], etc)– user can look in any direction at few given locations
• Image-morphing ([Wolberg90], [Beier92], etc)– linearly interpolated intermediate positions of features– input: two images and correspondences– output: metamorphosis of one image into other as
sequence of intermediate images
![Page 40: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/40.jpg)
40
Previous work limitations
• Panoramas ([Chen95], etc.)– no camera translations allowed
• Image morphing ([Wolberg90], [Beier92], etc.)– not shape-preserving– image morphing is also a morph of the object– to simulate rendering with morphing, the object
should be rigid when camera moves
![Page 41: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/41.jpg)
41
Overview
• Introduction• Image morphing• View morphing
– image pre-warping– image morphing– image post-warping
![Page 42: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/42.jpg)
42
Overview
• Introduction• Image morphing• View morphing
– image pre-warping– image morphing– image post-warping
![Page 43: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/43.jpg)
43
Image morphing1. Correspondences
![Page 44: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/44.jpg)
44
Image morphing1. Correspondences
![Page 45: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/45.jpg)
45
Image morphing1. Correspondences
![Page 46: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/46.jpg)
46
Image morphing1. Correspondences
![Page 47: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/47.jpg)
47
Image morphing1. Correspondences2. Linear
interpolation
nk PnkP
nkP
0)1(
P0 Pk Pn
frame 0 frame k frame n
![Page 48: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/48.jpg)
48
Image morphing
• Image morphing• not shape preserving
![Page 49: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/49.jpg)
49
Early IBR research
Soft watch at moment of first explosion – Salvador Dali 1954
![Page 50: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/50.jpg)
50
Overview
• Introduction• Image morphing• View morphing
– image pre-warping– image morphing– image post-warping
![Page 51: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/51.jpg)
51
Overview
• Introduction• Image morphing• View morphing
– image pre-warping– image morphing– image post-warping
![Page 52: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/52.jpg)
52
View morphing
• Shape preserving morph• Three step algorithm
1. Prewarp first and last images to parallel views2. Image morph between prewarped images3. Postwarp to interpolated view
![Page 53: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/53.jpg)
53
Step 1: prewarp to parallel views
• Parallel views– same image plane– image plane parallel to segment
connecting the two centers of projection• Prewarp
– compute parallel views I0p, Inp
– rotate I0 and In to parallel views
– prewarp corrs. (P0, Pn) -> (Pop, Pnp)
Cn
C0
I0
In
Inp
I0p
P
P0 Pn
P0p
Pnp
![Page 54: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/54.jpg)
54
Step 2: morph parallel images
• Shape preserving• Use prewarped correspondences• Interpolate Ck from C0 Cn
Cn
C0
I0
In
InpI0p
P
P0 Pn
P0p
Pnp
Ck
Pkp
![Page 55: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/55.jpg)
55
Step 3: Postwarping
• Postwarp morphed image– create intermediate view
• Ck is known• interpolate view
direction and tilt – rotate morphed image to
intermediate view
Cn
C0
I0
In
I0p
Ck
![Page 56: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/56.jpg)
56
View morphing
• View morphing• shape preserving
![Page 57: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/57.jpg)
57
Overview
• Introduction• Image morphing• View morphing, more details
– image pre-warping– image morphing– image post-warping
![Page 58: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/58.jpg)
58
Step 1: prewarp to parallel views
• Parallel views– use C0Cn for x (ap vector)
– use (a0 x b0) x (an x bn) as y (-bp)
– pick ap and bp to resemble a0 b0 as much as possible
– use same pixel size– use wider field of view
Cn
C0
I0
In
Inp
I0p
P
P0 Pn
P0p
Pnp
![Page 59: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/59.jpg)
59
Step 1: prewarp to parallel views
• prewarping using texture mapping– create polygon for image plane– consider it texture mapped with the image
itself– render the “scene” from prewarped view– if you go this path you will have to
implement clipping with the COP plane– you have texture mapping already
• alternative: prewarping using reprojection of rays
– look up all the rays of the prewarped view in the original viewCn
C0
I0
In
Inp
I0p
P
P0 Pn
P0p
Pnp
![Page 60: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/60.jpg)
60
Step 1: prewarp to parallel views
• prewarping correspondences– for all pairs of correspondence P0 Pn
• project P0 on I0p, computing P0p
• project Pn on Inp, computing Pnp
• prewarped correspondence is Pop Pnp
Cn
C0
I0
In
Inp
I0p
P
P0 Pn
P0p
Pnp
![Page 61: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/61.jpg)
61
Step 2: morph parallel images• Image morphing
– use prewarped correspondences to compute a correspondence for all pixels in I0p
– linearly interpolate I0p to intermediate positions– useful observation
• corresponding pixels are on same line in prewarped views
– preventing holes• use larger footprint (ex 2x2)• or linearly interpolate between consecutive
samples• or postprocess morphed image looking for
background pixels and replacing them with neighboring values
– visibility artifacts• collision of samples
– zbuffer on disparity• holes
– morph Inp to Ikp
– use additional viewsCn
C0
I0
In
InpI0p
P
P0 Pn
P0p
Pnp
Ck
Pkp
![Page 62: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/62.jpg)
62
Step 3: Postwarping
• create intermediate view– Ck is known– current view direction is a
linear interpolation of the start and end view directions
– current up vector is a linear interpolation of the start and end up vectors
• rotate morphed image to intermediate view– same as prewarping
Cn
C0
I0
In
I0p
Ck
![Page 63: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/63.jpg)
63
Image-Based Rendering by Warping
![Page 64: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/64.jpg)
64
Overview
• Introduction• Depth extraction methods• Reconstruction for IBRW• Visibility without depth• Sample selection
![Page 65: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/65.jpg)
65
Overview
• Introduction– comparison to other IBR methods– 3D warping equation– reconstruction
![Page 66: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/66.jpg)
66
IBR by Warping (IBRW)
• Images enhanced with per-pixel depth [McMillan95]
u1
v1P1
![Page 67: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/67.jpg)
67
IBR by Warping (IBRW)
C1
Pu1
v1
11
11
1111111 )(
PCPCw
wbvaucCP
P1
• 1/w1 also called generalized disparity
• another notation δ(u1, v1)
![Page 68: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/68.jpg)
68
IBR by Warping (IBRW)
C1
P
C2
u2
v2
u1
v1
2222222
1111111
)(
)(
wbvaucCP
wbvaucCP
P1
P2
![Page 69: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/69.jpg)
69
3D warping equations
C1
P
C2
u2
v2
u1
v1
),(),(),(),(
113413313231
1124123122212
113413313231
1114113112112
vuwvwuwwvuwvwuww
v
vuwvwuwwvuwvwuww
u
![Page 70: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/70.jpg)
70
A complete IBR method
• Façade system– (coarse) geometric model needed
• Panoramas– viewer confined to center of panorama
• View morphing– correspondences needed
• IBRW– rendering to arbitrary new views
![Page 71: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/71.jpg)
71
DeltaSphere - depth&color acquisition device
• Lars Nyland et al.
![Page 72: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/72.jpg)
72
![Page 73: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/73.jpg)
73
![Page 74: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/74.jpg)
74
![Page 75: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/75.jpg)
75
![Page 76: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/76.jpg)
76
Reconstructing by splatting
• Estimate shape and size of footprint of warped samples– expensive to do accurately– lower image quality if crudely approximated
• Samples are z-buffered
![Page 77: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/77.jpg)
77
Overview
• Introduction• Depth extraction methods• Reconstruction for IBRW• Visibility without depth• Sample selection
![Page 78: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/78.jpg)
78
Finding depth
![Page 79: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/79.jpg)
79
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 80: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/80.jpg)
80
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 81: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/81.jpg)
81
Depth from stereo
C1
P
C2
u2
v2
u1
v1 P1 P2
• two cameras with known parameters
• infer 3D location of point seen in both images
• sub problem: correspondences
• for a point seen in the left image, find its projection in the right image
![Page 82: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/82.jpg)
82
Depth from stereo: déjà-vu math
C1
P
C2
u2
v2
u1
v1
2222222
1111111
)(
)(
wbvaucCP
wbvaucCP
P1P2
• unknowns are w1 and w2
• overconstrained system
• the u2v2 coordinates of a point seen at u1v1 are constrained to an epipolar line
![Page 83: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/83.jpg)
83
Epipolar line
C1
P
C2
u2
v2
u1
v1P1
P2
• C1, C2, P1 define a plane
• P2 will be on that plane
• P2 is also on the image plane 2
• So P2 will be on the line defined by the two planes’ intersection
![Page 84: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/84.jpg)
84
Search for correspondences on epipolar line
C1
P
C2
u2
v2
u1
v1P1
P2
• Reduces the dimensionality of the search space
• Walk on epipolar segment rather than search in entire image
![Page 85: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/85.jpg)
85
Parallel views
C1
P
C2
u2
v1
u1
v1
• Preferred stereo configuration
• epipolar lines are horizontal, easy to search
![Page 86: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/86.jpg)
86
Parallel views
C1
P
C2
u2
v1
u1
v1
u1
• Limit search to epipolar segment
• from u2 = u1 (P is infinitely far away) to 0 (P is close)
![Page 87: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/87.jpg)
87
Depth precision analysis
C1
P
C2
u2
v1
u1
v1
• 1/z linear with disparity (u1 – u2)
• better depth resolution for nearby objects
• important to determine correspondences with subpixel accuracy
![Page 88: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/88.jpg)
88
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 89: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/89.jpg)
89
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 90: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/90.jpg)
90
Depth from stereo problem
• Correspondences are difficult to find• Structured light approach
– replace one camera with projector– project easily detectable patterns– establishing correspondences becomes a lot
easier
![Page 91: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/91.jpg)
91
Depth from structured light
C1
P
C2
u2
v2
u1
v1
2222222
1111111
)(
)(
wbvaucCP
wbvaucCP
P1 P2
• C1 is a projector
• Projects a pattern centered at u1v1
• Pattern center hits object scene at P
• Camera C2 sees pattern at u2v2, easy to find
• 3D location of P is determined
![Page 92: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/92.jpg)
![Page 93: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/93.jpg)
93
![Page 94: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/94.jpg)
94
Depth from structured light challenges
• Associated with using projectors– expensive, cannot be used outdoors, not
portable• Difficult to identify pattern
– I found a corner, which corner is it?• Invasive, change the color of the scene
– one could use invisible light, IR
![Page 95: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/95.jpg)
95
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 96: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/96.jpg)
96
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 97: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/97.jpg)
97
Depth of field
aperture
object
image
C F
F’
• Thin lenses
• rays through lens center (C) do not change direction
• rays parallel to optical axis go through focal point (F’)
![Page 98: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/98.jpg)
98
Depth of field
aperture
object
image plane
C F
F’
• For a given focal length, only objects that are at a certain depth are in focus
![Page 99: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/99.jpg)
99
Out of focus
aperture
object
image plane
C F
F’
• When object at different depth
• One point projects to several locations in the image
• Out of focus, blurred image
![Page 100: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/100.jpg)
100
Focusing
aperture
object
image plane
C F
F’
• Move lens to focus for new depth
• Relationship between focus and depth can be exploited to extract depth
![Page 101: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/101.jpg)
101
Determine z for points in focus
aperture
object
image plane
C F
F’
zfa
hh
fa i
hi
h
a f
z
![Page 102: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/102.jpg)
102
Depth from defocus
• Take images of a scene with various camera parameters
• Measuring defocus variation, infer range to objects
• Does not need to find the best focusing planes for the various objects
• Examples by Shree Nayar, Columbia U
![Page 103: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/103.jpg)
103
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 104: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/104.jpg)
104
Overview
• Depth from stereo• Depth from structured light• Depth from focus / defocus• Laser rangefinders
![Page 105: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/105.jpg)
105
Laser range finders
• Send a laser beam to measure the distance– like RADAR, measures time of flight
![Page 106: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/106.jpg)
106
DeltaSphere - depth&color acquisition device
• Lars Nyland et al.
courtesy 3rd Tech Inc.
![Page 107: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/107.jpg)
107
• 300o x 300o panorama• this is the reflected light
![Page 108: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/108.jpg)
108
• 300o x 300o panorama• this is the range light
![Page 109: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/109.jpg)
109
courtesy 3rd Tech Inc.
spherical range panoramas
planar re-projection
![Page 110: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/110.jpg)
110
courtesy 3rd Tech Inc.
Jeep – one scan
![Page 111: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/111.jpg)
111
courtesy 3rd Tech Inc.
Jeep – one scan
![Page 112: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/112.jpg)
112
courtesy 3rd Tech Inc.
Complete Jeep model
![Page 113: Image Based Rendering](https://reader035.fdocuments.us/reader035/viewer/2022062521/568155a4550346895dc38252/html5/thumbnails/113.jpg)
113