104/21/23 07:42

Graphics II 91.547

Image Based Rendering

Session 11

204/21/23 07:42

A Rendering Taxonomy

304/21/23 07:42

The Plenoptic Function

p P V V V tx y z ( , , , , , , )

“… the pencil of rays visible from any point in space, at any time, and over any range of wavelengths”

Given a set of discrete samples (complete or incomplete) from the plenoptic function, the goal of image-based rendering is to generate a continuous representation of that function.

404/21/23 07:42

Movie Map(Lippman 1980)

V V Vx y z, , , , Find Nearest

Sample MovieStorage

Image

504/21/23 07:42

Taxonomy of “Virtual Camera” Movement(Chen et al. 1995)

0 Camera Rotation- Camera fixed at a particular location- Three rotational degrees of freedom

=Pitch (up and down)=Yaw (about vertical axis)=Roll (about camera axis)

0 Object Rotation- Camera always pointing at center of object- Viewpoint constrained to move over surface of sphere- Three angular degrees of freedom

0 Camera movement- Viewpoint unconstrained- Viewing direction unconstrained

604/21/23 07:42

Environment MapsMap Geometries

Cube Sphere Cylinder

704/21/23 07:42

Quick Time VRTM

(Chen 1995)

2500 Pixels

768 Pixels

2500 x 768 = 1.9 G Pixels x 3 B/pixel = 5.8 GB10:1 compression 500 MB/panorama

804/21/23 07:42

Image Distortion fromCylindrical Environment Map

Projection Plane

CylindricalEnvironmentMap

Pre-warpedProjection ontoPlane

904/21/23 07:42

Quick Time VRImage Warping for Correct Perspective View

1004/21/23 07:42

Quick Time VRPanoramic Display Process

Compressed Tiles

Visible Tiles

CD ROM orHard Disk

Main MemoryCompressed

TilesCache

VisibleRegion

DisplayWindow

Warp

Decompress

OffscreenBuffer

1104/21/23 07:42

Quick Time VRAccomplishing (Limited) Camera Motion

1204/21/23 07:42

Accomplishing Camera MotionGreene&Kass (1993 Apple Tech Doc.)

0 Regular 3-D lattice of cubic environment maps0 Each environment map is a z-buffered rendering from a

discrete viewpoint0 Image from a new viewpoint is generated by re-sampling the

environment map0 Re-sampling involves rendering the pixels in the environment

maps as 3-D polygons from the new viewpoint0 Rendering time proportional to the environment map

resolution but independent of scene complexity0 Not suitable for real-time walkthrough performance on typical

desktop computers (especially in 1993!)

1304/21/23 07:42

Alternative approach:Work entirely in Image Space

0 Sequence of images from closely spaced viewpoints is highly coherent

0 Depends upon the ability to establish a pixel-by-pixel correspondence between adjacent images- Can be computed if range data and camera parameters

are known (true for rendered images)- For natural images, there are several techniques including

manual user intervention0 Pairwise correspondence between two images can be stored

as a pair of morph maps- Bi-directional maps required because of possible many to

one and one to many pixel correspondences0 Can be represented by graph data structure where nodes are

images and arcs are bi-directional morph maps

1404/21/23 07:42

N-Dimensional Graph Data Structure

Image Image Image Image Image

Image Image Image Image Image

Image Image Image Image Image

Bi-directional Morph Maps

1504/21/23 07:42

Simple View Interpolation

Reference Image 1 Reference Image 2

CorrespondingPixels

Morph maps

1604/21/23 07:42

Image Overlap orImage Folding

P1

P2

ReferenceView

InterpolatedView

1704/21/23 07:42

Image Holes orImage Stretching

InterpolatedView

ReferenceView

P1P2

1804/21/23 07:42

Example of Hole Region

Viewpoint 1 Viewpoint 2

1904/21/23 07:42

Example of Hole RegionMinimizing by Closely Spaced Viewpoints

Viewpoint 1 Viewpoint 2

2004/21/23 07:42

Source Image Viewed from Camera Moved to the Right

Ref. View 1

Ref. View 2

2104/21/23 07:42

Offset Vectors for Camera Motion Morph Map

2204/21/23 07:42

Locus of Morph Map for MotionParallel to Image Plane and Floor

2304/21/23 07:42

Distortion of Intermediate Images withLinear Warp

Linear path of one feature

2404/21/23 07:42

Morphing Parallel Views

Reference image

Reference image

Interpolated image

V0

V1

Ci

V V V

p p p

p V

i 0 1

i 0 1

i i

( )

( )

1

1

s s

s s

P

2504/21/23 07:42

View Interpolation:The Algorithm

I0 I1

I0 ' I1 'Is '

Is

1 1

2 2

3

2604/21/23 07:42

Example 1 of calculated intermediate images

Reference Image 1 Reference Image 2

Intermediate Views

2704/21/23 07:42

Example 2 of calculated intermediate images

Reference Image 1 Reference Image 2Interpolated Image

2804/21/23 07:42

Multiple-Center-of-Projection Images(Rademacher&Bishop 1998)

0 Information from a set of viewpoints stored in a single image0 Features

- Greater connectivity information compared with collections of standard images

- Greater flexibility in the acquisition of image-based datasets, e.g. sampling different portions of the scene at different resolutions

2904/21/23 07:42

Multiple-Center-of-Projection ImagesDefinition

0 A multiple-center-of-projection image consists of a two-dimensional image and a parameterized set of cameras meeting the following conditions:- The cameras must lie on either a continuous curve or a

continuous surface- Each pixel is acquired by a single camera- Viewing rays vary continuously across neighboring pixels- Two neighboring pixels must either correspond to the

same camera or to neighboring cameras- Each pixel contains range information

3004/21/23 07:42

MCOP Image

3104/21/23 07:42

Strip Camera used for Capture of Real MCOP Images

3204/21/23 07:42

Camera Path in Capturing MCOP Image of Castle

3304/21/23 07:42

Image Plane for Camera Motion

3404/21/23 07:42

Resulting 1000 x 500 MCOP Image

3504/21/23 07:42

Reprojection

Camera model, stored per column:

Center of projection

Vector from to image plane origin

Horizontal axis of viewing plane

Vertical axis of viewing plane

Disparity = distance from to the image planedivided by distance from to the pixel’s world space point

Ci

CiOi

U i

Vi

ijCi

Ci

x

y

z

i

j

C

C

Cij

ix ix ix

iy iy iy

iz iz iz

ix

iy

iz

1

1

U V O

U V O

U V O

ReprojectionFormula:

3604/21/23 07:42

View of Castle Reconstructed from MCOP Image

3704/21/23 07:42

AnotherView of Castle Reconstructed from MCOP Image

3804/21/23 07:42

Lumigraphs

0 Lumigraph = a representation of the light resulting from a scene

0 Limited data representation of the plenoptic function0 Generated from multiple images and camera “poses”0 Rendering: Image = Lumigraph + Camera Model0 Special case of 4D Light Field (Levoy, Hanrahan)

3904/21/23 07:42

What is a Lumigraph?

For all points on the surrounding surface,For all directions,

The color intensity of the ray.

Assumption: We are outside a convex hull containingthe objects

4004/21/23 07:42

Parameterization of the Lumigraph

Images from Steven Gortler, SIGGRAPH 1999

4104/21/23 07:42

Building the Lumigraph

4204/21/23 07:42

Approximating the LumigraphWith Discrete Samples

4304/21/23 07:42

Views of a Light Field (Lumigraph)

Levoy & Hanrahan, Light Field Rendering, Computer Graphics