Rendering With Coherent Layers

Jed LengyelJed Lengyel John SnyderJohn Snyder

Microsoft ResearchMicrosoft Research

Jed LengyelJed Lengyel John SnyderJohn Snyder

Microsoft ResearchMicrosoft Research

SIGGRAPH 97

Traditional Pipeline

Quality

3DScene

RendererOutputImage

Quality

3DScene

RendererOutputImage

Renders 3D scene to an image.Renders 3D scene to an image. Renders 3D scene to an image.Renders 3D scene to an image.

Traditional Pipeline

Quality

3DScene

RendererOutputImage

Quality

3DScene

RendererOutputImage

Traditional Pipeline

Quality

3DScene

RendererOutputImage

Quality

3DScene

RendererOutputImage

ProblemsProblems

• Does not exploit temporal coherenceDoes not exploit temporal coherence

• Spatial and temporal resolutions are globalSpatial and temporal resolutions are global

• Limited integration of 2D elements in 3DLimited integration of 2D elements in 3D

ProblemsProblems

• Does not exploit temporal coherenceDoes not exploit temporal coherence

• Spatial and temporal resolutions are globalSpatial and temporal resolutions are global

• Limited integration of 2D elements in 3DLimited integration of 2D elements in 3D

Traditional Pipeline

Quality

3DScene

RendererOutputImage

Quality

3DScene

RendererOutputImage

Traditional Pipeline

Quality

3DScene

RendererOutputImage

Quality

3DScene

RendererOutputImage

How can we improve this?How can we improve this?How can we improve this?How can we improve this?

Layered Pipeline

Quality

3DScene

RendererOutputImage

2D ImageTransform

Quality

3DScene

RendererOutputImage

2D ImageTransform

Add a 2D image warp.Add a 2D image warp.Add a 2D image warp.Add a 2D image warp.

Layered Pipeline

SpriteSprite

Quality

3DScene

RendererOutputImage

2D ImageTransform

Quality

3DScene

RendererOutputImage

2D ImageTransform

A sprite is a warped image with A sprite is a warped image with alpha.alpha.A sprite is a warped image with A sprite is a warped image with alpha.alpha.

LayerLayer

Layered Pipeline

Quality

3DScene

Renderer Sprite

Quality

3DScene

Renderer Sprite

Each layer produces a sprite.Each layer produces a sprite.Each layer produces a sprite.Each layer produces a sprite.

3DScene

Com

posi

tor

OutputImage

Split

ter

Layering Order

Layer 0

Layer 1

Layer N-1

3DScene

Com

posi

tor

OutputImage

Split

ter

Layering Order

Layer 0

Layer 1

Layer N-1

Layered PipelineThe scene is factored into layers.The scene is factored into layers.The scene is factored into layers.The scene is factored into layers.

Flyby Film

Spatial Resolution

Previous Work• Image-caching Image-caching [Shade96, Schaufler96][Shade96, Schaufler96]

• Composition architectures (with z per pixel) Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97][Duff85, Molnar92, Regan94, Mark97]

• Image-based rendering Image-based rendering [Chen93, Chen95, [Chen93, Chen95, McMillan95]McMillan95]

• Regulation Regulation [Funkhouser93, Maciel95][Funkhouser93, Maciel95]

• Shading factorization Shading factorization [Cook84, Hanrahan90, [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Segal92, Dorsey95, Guenter95, Meier96]

• Image-caching Image-caching [Shade96, Schaufler96][Shade96, Schaufler96]

• Composition architectures (with z per pixel) Composition architectures (with z per pixel) [Duff85, Molnar92, Regan94, Mark97][Duff85, Molnar92, Regan94, Mark97]

• Image-based rendering Image-based rendering [Chen93, Chen95, [Chen93, Chen95, McMillan95]McMillan95]

• Regulation Regulation [Funkhouser93, Maciel95][Funkhouser93, Maciel95]

• Shading factorization Shading factorization [Cook84, Hanrahan90, [Cook84, Hanrahan90, Segal92, Dorsey95, Guenter95, Meier96]Segal92, Dorsey95, Guenter95, Meier96]

2D Image TransformT is 3D transform to screen T is 3D transform to screen coordinates.coordinates.T is 3D transform to screen T is 3D transform to screen coordinates.coordinates.

3D Shape

T

Compositor

3D ShapeA

A -1 T

T

Sprite

2D Image TransformExtra degrees of freedomExtra degrees of freedomExtra degrees of freedomExtra degrees of freedom

2D Image TransformComposition maps to same screen Composition maps to same screen point.point.Composition maps to same screen Composition maps to same screen point.point.

Compositor

3D ShapeA

A -1 T

T

pq

p'

Sprite

2D Image TransformSpatial resolutionSpatial resolutionSpatial resolutionSpatial resolution

Compositor

3D ShapeA

A -1 T

C

C -1 T

Sprite C

Sprite A

2D Image TransformTemporal resolutionTemporal resolutionTemporal resolutionTemporal resolution

3D Shape

A 0

SpriteFrame

0

Framek-1

A k-1

A 0-1 T

Factoring Geometry

Factoring GeometryGoodGoodGoodGood

Factoring GeometryGoodGoodGoodGood

Factoring GeometryGoodGoodGoodGood

Factoring GeometryGoodGoodGoodGood

Factoring GeometryPoorPoorPoorPoor

Factoring GeometryPoorPoorPoorPoor

Factoring GeometryPoor Poor Poor Poor

Factoring GeometryPoorPoorPoorPoor

Factoring GeometryPoorPoorPoorPoor

Depth Sorting of Layers

Depth sorting in software is Depth sorting in software is

effective.effective.

• Relatively small number of primitivesRelatively small number of primitives

• Exploits temporal coherenceExploits temporal coherence

Depth sorting in software is Depth sorting in software is

effective.effective.

• Relatively small number of primitivesRelatively small number of primitives

• Exploits temporal coherenceExploits temporal coherence

Factoring Shading

Multipass Shading

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

[Segal92][Segal92][Segal92][Segal92]

Multipass Shading

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

Factoring Shading

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

Com

posi

tor

OutputImage

Full

Shadow

ShadowMap

Reflection

ReflectionMap

3DScene

*+

Com

posi

tor

3D Shape

A

A -1 T

Sprite Display

Warp Choice

• Accuracy of image interpolation through timeAccuracy of image interpolation through time

• Efficient hardware implementationEfficient hardware implementation

• Ease of computing the warp parametersEase of computing the warp parameters

• Accuracy of image interpolation through timeAccuracy of image interpolation through time

• Efficient hardware implementationEfficient hardware implementation

• Ease of computing the warp parametersEase of computing the warp parameters

Characteristic PointsTrack 3D motion projected to 2DTrack 3D motion projected to 2DTrack 3D motion projected to 2DTrack 3D motion projected to 2D

3D Shape Screen Coordinates

T

3D Shape Screen Coordinates

T

Warp Calculation

3D Shape

Frame 0 Frame 1

W

T 1

T 0

3D Shape

Frame 0 Frame 1

W

T 1

T 0

Characteristic points are matchedCharacteristic points are matchedCharacteristic points are matchedCharacteristic points are matched

Warp Calculation

Frame 0 Frame 1

W

Frame 0 Frame 1

W

W PW P00 = P= P11W PW P00 = P= P11

Warp Calculation

Frame 0 Frame 1

W

Frame 0 Frame 1

W

W PW P00 = P= P11W PW P00 = P= P11

Warps Considered• Pure translationPure translation

• Translation with isotropic scaleTranslation with isotropic scale

• Translation with independent scale in x and Translation with independent scale in x and

yy

• AffineAffine

• PerspectivePerspective

• Pure translationPure translation

• Translation with isotropic scaleTranslation with isotropic scale

• Translation with independent scale in x and Translation with independent scale in x and

yy

• AffineAffine

• PerspectivePerspective

Warp Comparison

Image InterpolationWarping intermediate images is Warping intermediate images is effectiveeffective

• Triple- or quadruple-framingTriple- or quadruple-framing

• Warp calculated with a small set of pointsWarp calculated with a small set of points

• Decouples 3D rendering from displayDecouples 3D rendering from display

Warping intermediate images is Warping intermediate images is effectiveeffective

• Triple- or quadruple-framingTriple- or quadruple-framing

• Warp calculated with a small set of pointsWarp calculated with a small set of points

• Decouples 3D rendering from displayDecouples 3D rendering from display

Image Interpolation

3DScene

Com

posi

tor

OutputImage

Split

ter

Layering Order

Layer 0

Layer 1

Layer N-1

3DScene

Com

posi

tor

OutputImage

Split

ter

Layering Order

Layer 0

Layer 1

Layer N-1

RegulationSet independent layer quality Set independent layer quality parametersparametersSet independent layer quality Set independent layer quality parametersparameters

Maximize fidelity, balance Maximize fidelity, balance resourcesresourcesMaximize fidelity, balance Maximize fidelity, balance resourcesresources

Regulator

Layer 1Quality 1 Fidelity 1

Layer 0

Layer N-1

Quality 0

Quality N-1

Fidelity 0

Fidelity N-1

Regulator

Layer 1Quality 1 Fidelity 1

Layer 0

Layer N-1

Quality 0

Quality N-1

Fidelity 0

Fidelity N-1

Regulator

FiducialsMeasure fidelity of approximationMeasure fidelity of approximation

• GeometricGeometric

• PhotometricPhotometric

• SamplingSampling

• VisibilityVisibility

Measure fidelity of approximationMeasure fidelity of approximation

• GeometricGeometric

• PhotometricPhotometric

• SamplingSampling

• VisibilityVisibility

Geometric FiducialCompares warped and current Compares warped and current pointspointsCompares warped and current Compares warped and current pointspoints

Photometric FiducialSamples lighting at characteristic Samples lighting at characteristic pointspointsSamples lighting at characteristic Samples lighting at characteristic pointspoints

Frame 0 Frame 1Frame 0 Frame 1

Photometric Fiducial 2Measures change in light positionMeasures change in light positionMeasures change in light positionMeasures change in light position

Sampling FiducialMeasures distortion of image Measures distortion of image samplessamplesMeasures distortion of image Measures distortion of image samplessamples

A 0

A i

A 0

A i

Visibility Fiducial Counts back-to-front transitionsCounts back-to-front transitions Counts back-to-front transitionsCounts back-to-front transitions

Frame 0Frame 0 Frame 1Frame 1

Barnyard Example

ConclusionLayers are good for real-time 3D.Layers are good for real-time 3D.

• Effective using affine image warpEffective using affine image warp

• Exploit temporal coherenceExploit temporal coherence

• Factor shadingFactor shading

• Allow precise targeting of resourcesAllow precise targeting of resources

• 3x-10x improvement3x-10x improvement

Layers are good for real-time 3D.Layers are good for real-time 3D.

• Effective using affine image warpEffective using affine image warp

• Exploit temporal coherenceExploit temporal coherence

• Factor shadingFactor shading

• Allow precise targeting of resourcesAllow precise targeting of resources

• 3x-10x improvement3x-10x improvement