Advanced Computer Graphics Advanced Computer Graphics (Spring 2005) (Spring 2005)

COMS 4162, Lecture 21: Image-Based Rendering

Ravi Ramamoorthi

http://www.cs.columbia.edu/~cs4162

To Do / MotivationTo Do / Motivation

Work hard on assignment 4

This last series of lectures covers (at a high level) some more advanced topics and areas of current research interest in modern rendering

Course OutlineCourse Outline

3D Graphics Pipeline

Rendering(Creating, shading images from geometry, lighting, materials)

Modeling(Creating 3D Geometry)

Next few slides courtesy Paul Debevec; SIGGRAPH 99 course notes

Image-Based Modeling and RenderingImage-Based Modeling and Rendering

Generate new views of a scene directly from existing views

“Pure” IBR (such as lightfields): no geometric model of scene

Other IBR techniques try to obtain higher quality with less storage by building a model

IBR: Pros and ConsIBR: Pros and Cons

Advantages Easy to capture images: photorealistic by definition Simple, universal representation Often bypass geometry estimation? Independent of scene complexity?

Disadvantages WYSIWYG but also WYSIAYG Explosion of data as flexibility increased Often discards intrinsic structure of model?

IBR: A brief historyIBR: A brief history

Texture maps, bump maps, env. maps [70s]

Poggio et al. MIT: Faces, image-based analysis/synthesis

Modern Era Chen and Williams 93, View Interpolation [Images with depth] Chen 95 Quicktime VR [Images from many viewpoints] McMillan and Bishop 95 Plenoptic Modeling [Images w disparity] Gortler et al, Levoy and Hanrahan 96 Light Fields [4D] Shade et al. 98 Layered Depth Images [2.5D] Debevec et al. 00 Reflectance Field [4D] Inverse rendering methods (Sato,Yu,Marschner,Boivin,…)

Fundamentally, sampled representations in graphics

OutlineOutline

Overview of IBR

Basic approaches Image Warping Light Fields Survey of some recent work

Warping slides courtesy Leonard McMillan

OutlineOutline

Overview of IBR

Basic approaches Image Warping

[2D + depth. Requires correspondence/disparity] Light Fields [4D] Survey of some recent work

Plenoptic FunctionPlenoptic Function

L(x,y,z,,,t,)

Captures all light flow in a scene to/from any point (x,y,z), in any direction (,), at any time (t), at any frequency ()

Enough information toconstruct any imageof the scene at any time

(x,y,z)(x,y,z)((,,))

[Funkhouser][Funkhouser]

Plenoptic Function SimplificationsPlenoptic Function Simplifications

Represent color as RGB: eliminate

Static scenes: ignore dependence on t

7D 3 5D

Lumigraph PostprocessingLumigraph Postprocessing

Obtain rough geometric model Chroma keying (blue screen) to extract silhouettes Octree-based space carving

Resample images to two-plane parameterization

Lumigraph RenderingLumigraph Rendering

Use rough depth information to improve rendering quality

Lumigraph RenderingLumigraph Rendering

Use rough depth information to improve rendering quality

Lumigraph RenderingLumigraph Rendering

Without usinggeometry

Using approximategeometry

Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering

Further enhancement of lumigraphs:do not use two-plane parameterization

Store original pictures: no resampling

Hand-held camera, moved around an environment

Unstructured Lumigraph RenderingUnstructured Lumigraph Rendering

To reconstruct views, assign penalty to each original ray Distance to desired ray, using

approximate geometry Resolution Feather near edges of image

Construct “camera blending field”

Render using texture mapping

Unstructured Lumigraph Unstructured Lumigraph RenderingRendering

Blending field Rendering

OutlineOutline

Overview of IBR

Basic approaches Image Warping

[2D + depth. Requires correspondence/disparity] Light Fields [4D] Survey of some recent work

LDIsLDIs

Layered depth images [Shade et al. 98]

Geometry

Camera

Slide from Agrawala, Ramamoorthi, Heirich, Moll, SIGGRAPH 2000

LDIsLDIs

Layered depth images [Shade et al. 98]

LDI

LDIsLDIs

Layered depth images [Shade et al. 98]

LDI

(Depth, Color)

Surface Light FieldsSurface Light Fields

Miller 98, Nishino 99, Wood 00

Reflected light field (lumisphere) on surface

Explicit geometry as against light fields. Easier compress

Acquiring Reflectance Field of Human Acquiring Reflectance Field of Human Face [Debevec et al. SIGGRAPH 00]Face [Debevec et al. SIGGRAPH 00]

Illuminate subject from many incident directions

Example ImagesExample Images

Images from Debevec et al. 00

Conclusion (my views)Conclusion (my views)

Real issue is compactness/flexibility vs. rendering speed

IBR is use of sampled representations. Easy to interpolate, fast to render. If samples images, easy to acquire.

Of course, for this course, some pretty fancy precomputed algorithms (because we want to handle complex lighting that changes)

IBR in pure form not really practical WYSIAYG Explosion as increase dimensions (8D transfer function) Ultimately, compression, flexibility needs geometry/materials But lots of recent work (some in course) begins to correct these issues

Right question is tradeoff compactness/efficiency Factored representations Understand sampling rates and reconstruction