Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple...
Transcript of Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple...
![Page 1: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/1.jpg)
Marc Levoy
Image-based modeling (IBM)and image-based rendering (IBR)
CS 248 - Introduction to Computer GraphicsAutumn quarter, 2005
Slides for December 8 lecture
![Page 2: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/2.jpg)
Marc Levoy
modeling animation rendering
The graphics pipeline
![Page 3: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/3.jpg)
Marc Levoy
modeling animation rendering
3Dscanning
motioncapture
image-basedrendering
The graphics pipelinethe traditional pipeline
the new pipeline?
![Page 4: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/4.jpg)
Marc Levoy
IBM / IBR
“The study of image-based modelingand rendering is the study of
sampled representations of geometry.”
![Page 5: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/5.jpg)
Marc Levoy
Image-based representations:the classics
3D– model + texture/reflectance map [Blinn78]
– model + displacement map [Cook84]
– volume rendering [Levoy87, Drebin88]
2D + Z– range images [Binford73]
– disparity maps [vision literature]
2.5D– sprites [vis-sim, games]
n 2D– epipolar plane images [Bolles87]
– movie maps [Lippman78]
2D– environment maps, a.k.a. panoramas [19th century]less
geometry
more geometry
![Page 6: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/6.jpg)
Marc Levoy
Recent additionsfull model
– view-dependent textures[Debevec96]
– surface light fields [Wood00]
– Lumigraphs [Gortler96]
sets of range images– view interpolation [Chen93, McMillan95, Mark97]
– layered depth images [Shade98]
– relief textures [Oliveira00]
feature correspondences– plenoptic editing [Seitz98, Dorsey01]
camera pose– image caching [Schaufler96, Shade96]
– sprites + warps [Lengyel97]
– light fields [Levoy96]
no model– outward-looking QTVR [Chen95]
less geometry
more geometry
![Page 7: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/7.jpg)
Marc Levoy
Rangefinding technologies
• passive– shape from stereo– shape from focus– shape from motion, etc.
• active– texture-assisted shape-from-X– triangulation using structured-light– time-of-flight
![Page 8: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/8.jpg)
Marc Levoy
Laser triangulation rangefinding
![Page 9: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/9.jpg)
single scan of St. Matthew 1 mm
![Page 10: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/10.jpg)
Marc Levoy
Post-processing pipeline
• steps1. aligning the scans2. combining aligned scans3. filling holes
![Page 11: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/11.jpg)
Marc Levoy
Digitizing the statues of Michelangelo using laser scanners
• 480 individually aimed scans• 2 billion polygons• 7,000 color images• 30 nights of scanning• 22 people
![Page 12: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/12.jpg)
![Page 13: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/13.jpg)
![Page 14: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/14.jpg)
Marc Levoy
Replica of Michelangelo’s David(20 cm tall)
![Page 15: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/15.jpg)
Solving the jigsaw puzzleof the Forma Urbis Romae
![Page 16: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/16.jpg)
Marc Levoy
The puzzle as it now stands
![Page 17: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/17.jpg)
Marc Levoy
Clues for solving the puzzle
• incised lines• incision characteristics• marble veining• fragment thickness• shapes of fractured surfaces• rough / smooth bottom surface• straight sides, indicating slab boundaries• location and shapes of clamp holes• the wall: slab layout, clamp holes, stucco• archaeological evidence
![Page 18: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/18.jpg)
Matching incised lines
fragment 156
fragment 134
fragment 167
![Page 19: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/19.jpg)
fragment 156
fragment 134
fragment 167
![Page 20: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/20.jpg)
Marc Levoy
Geometry-based versusimage-based rendering
geometry images
flythrough of scene
conceptual world real world
rendering
computervisiongeometry-based
renderingimage-based
rendering
modelconstruction
imageacquisition
flythrough of scene
![Page 21: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/21.jpg)
Marc Levoy
Shortcutting thevision/graphics pipeline
geometry
views
real world
graphicspipeline
image-basedrendering
visionpipeline
(from M. Cohen)
![Page 22: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/22.jpg)
Marc Levoy
Apple QuickTime VR[Chen, Siggraph ’95]
• outward-looking– panoramic views taken at regularly spaced points
• inward-looking– views taken at points on the surface of a sphere
![Page 23: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/23.jpg)
Marc Levoy
View interpolationfrom a single view
1. Render object2. Convert Z-buffer to range image3. Tesselate to create polygon mesh
4. Re-render from new viewpoint5. Use depths to resolve overlaps
Q. How to fill in holes?
![Page 24: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/24.jpg)
Marc Levoy
View interpolationfrom multiple views
1. Render object from multiple viewpoints2. Convert Z-buffers to range images3. Tesselate to create multiple meshes
4. Re-render from new viewpoint5. Use depths to resolve overlaps6. Use multiple views to fill in holes
![Page 25: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/25.jpg)
Marc Levoy
Post-rendering 3D warping[Mark et al., I3D97]
• render at low frame rate• interpolate to real-time frame rate
– interpolate observer viewpoint using B-Spline– convert reference images to polygon meshes– warp meshes to interpolated viewpoint– composite by Z-buffer comparison and conditional write
![Page 26: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/26.jpg)
Marc Levoy
Results
• rendered at 5 fps, interpolated to 30 fps• live system requires reliable motion prediction
– tradeoff between accuracy and latency
• fails on specular objects
![Page 27: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/27.jpg)
Marc Levoy
Image caching[Shade et al., SIGGRAPH 1996]
• precompute BSP tree of scene (2D in this case)• for first observer position
– draw nearby nodes (yellow) as geometry– render distant nodes (red) to RGB images (black)– composite images together
• as observer moves– if disparity exceeds a threshold, rerender image
![Page 28: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/28.jpg)
Marc Levoy
Light field rendering[Levoy & Hanrahan, SIGGRAPH 1996]
• must stay outside convex hull of the object• like rebinning in computed tomography
![Page 29: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/29.jpg)
Marc Levoy
The plenoptic function
Radiance as a function of position and directionin a static scene with fixed illumination
• for general scenes 5D function
L ( x, y, z, )
• in free space 4D function“ the (scalar) light field”
![Page 30: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/30.jpg)
Marc Levoy
The free-space assumption
• applications for free-space light fields– flying around a compact object– flying through an uncluttered environment
![Page 31: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/31.jpg)
Some candidate parameterizations
Point-on-plane + direction
L ( x, y, )
• convenient for measuring luminaires
x
y
![Page 32: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/32.jpg)
More parameterizations
Chords of a sphere
L ( , )
• convenient for spherical gantry• facilitates uniform sampling
![Page 33: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/33.jpg)
Two planes (“light slab”)
L ( u, v, s, t )
• uses projective geometry– fast incremental display algorithms
u
v
s
t
![Page 34: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/34.jpg)
Marc Levoy
Creating a light field
• off-axis (sheared) perspective views
![Page 35: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/35.jpg)
Marc Levoy
A light field is an array of images
![Page 36: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/36.jpg)
Marc Levoy
Displaying a light field
foreach x,ycompute u,v,s,tI(x,y) = L(u,v,s,t)
![Page 37: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/37.jpg)
Marc Levoy
Devices for capturing light fields:Stanford Multi-Camera Array
• cameras closely packed– high-X imaging– synthetic aperture photography
• cameras widely spaced– video light fields– new computer vision algorithms
![Page 38: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/38.jpg)
Marc Levoy
The BRDF kaleidoscope[Han et al., SIGGRAPH 2003]
• discrete number of views• hard to capture grazing angles• uniformity?
![Page 39: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/39.jpg)
Marc Levoy
Light field morphing[Zhang et al., SIGGRAPH 2002]
• feature correspondences = 3D model
UI for specifying feature polygonsand their correspondences
sample morph
![Page 40: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/40.jpg)
Marc Levoy
Autostereoscopic display of light fields[Isaksen et al., SIGGRAPH 2000]
• image is at focal distance of lenslet collimated rays• spatial resolution ~ # of lenslets in the array• angular resolution ~ # of pixels behind each lenslet• each eye sees a different sets of pixels stereo
![Page 41: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/41.jpg)
Marc Levoy
End-to-end 3D television[Matusik et al., SIGGRAPH 2005]
• 16 cameras, 16 video projectors, lenticular lens array• spatial resolution ~ # of pixels in a camera and projector• angular resolution ~ # of cameras and projectors• horizontal parallax only
![Page 42: Image-based modeling (IBM) and image-based rendering (IBR) · 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes](https://reader036.fdocuments.us/reader036/viewer/2022062606/6005b80967885f6f2049d10c/html5/thumbnails/42.jpg)
Marc Levoy
Why didn’t IBR take over the world?
• warping and rendering range images is slow– pixel-sized triangles are inefficient– just as many pixels need to be touched as in normal rendering
• arms race against improvements in 3D rendering– level of detail (LOD)– culling techniques– hierarchical Z-buffer– etc.
• visual artifacts are objectionable– not small and homogeneous like 3D rendering artifacts