Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

29
Interactive Interactive Deformation of Deformation of Light Fields Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy

Transcript of Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Page 1: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Interactive Deformation Interactive Deformation of Light Fieldsof Light Fields

Billy Chen

Eyal Ofek

Heung-Yeung Shum

Marc Levoy

Page 2: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Image-based renderingImage-based rendering

Levoy et al., 1996 Shum et al., 1999

Zitnick et al., 2004 Uyttendaele et al., 2004

Page 3: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Previous workPrevious work

Seitz, S., Kutulakos, K. N. “Plenoptic image editing,” In Proc. ICCV 1998.

Zhang, Z., Wang, L., Guo, B., Shum, H.-Y, “Feature-based light field morphing,” In Proc. SIGGRAPH 2002.

Weyrich, T., Pfister, H., Gross, M., “Rendering deformable surface reflectance fields,” In IEEE Transactions on Computer Graphics and Visualization.

Page 4: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Light field deformationLight field deformation

Page 5: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

OutlineOutline

Coaxial light fields Framework for deformation

Splitting Deforming Rendering

Results

Page 6: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

OutlineOutline

Coaxial light fields Framework for deformation

Splitting Deforming Rendering

Results

Page 7: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Coaxial light fieldsCoaxial light fields

… …

Page 8: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.
Page 9: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.
Page 10: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

OutlineOutline

Coaxial light fields Framework for deformation

Splitting Deforming Rendering

Results

Page 11: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Splitting a light fieldSplitting a light field

A sub-light field consists of a bundle of rays a deformation box

u

s

Page 12: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Sub-light field exampleSub-light field example

Page 13: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Projector-based light field Projector-based light field segmentationsegmentation

Page 14: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Page 15: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.
Page 16: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

OutlineOutline

Coaxial light fields Framework for deformation

Splitting Deforming Rendering

Results

Page 17: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

DeformationDeformation

Page 18: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Ray warpingRay warping

• preserves mapping specified by the animator• preserves C0

continuity of points• maps lines to lines

Page 19: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Inverse view ray warpingInverse view ray warping

Page 20: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Inverse view ray warpingInverse view ray warping

Page 21: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Hardware accelerated warpingHardware accelerated warping

front id map back id map deformation

• 256 x 256 resolution texture map• anisoptric filter for mip-mapped texture• 64 x 64 triangles• real-time with a NVIDIA GeForce 3

Page 22: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

OutlineOutline

Coaxial light fields Framework for deformation

Splitting Deforming Rendering

Results

Page 23: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

For each camera pixel1. Segment ray2. Warp ray3. Render ray4. Mask ray

Rendering and visibilityRendering and visibility

Page 24: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

ResultsResults

Page 25: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

ConclusionConclusion

A pipeline for deforming light fields Light field segmentation Ray warping Rendering multiple light fields

Coaxial light fields Enables interactive deformation of photo-

realistic objects Applications: photo-realistic rapid prototyping,

games

Page 26: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Thank youThank you

Reviewers NSF (IIS-0219856-001) DARPA (NBCH-1030009) Elizabeth Chen, Vaibhav Vaish, Gaurav Garg

Page 27: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Failure casesFailure cases

Page 28: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

Video of acquisition processVideo of acquisition process

Page 29: Interactive Deformation of Light Fields Billy Chen Eyal Ofek Heung-Yeung Shum Marc Levoy.

DeformationDeformationand compositingand compositing