Interactive Rendering using the Render Cache

Bruce Walter, George DrettakisiMAGIS*-GRAVIR/IMAG-INRIA

Steven ParkerUniversity of Utah

*iMAGIS is a joint project of CNRS/IMAG/INRIA and UJF

Motivation

Goal: Interactive rendering

Ray tracing Path tracing

Motivation

High-quality renderers• Pixel based

- Ray tracing, path tracing, etc.

• Wide range of lighting effects- Reflection, refraction, global illumination, etc.

• Too slow for interactive use- Many seconds per image

- Often used only for final images

- Alternate renderers used for interactive editing

Motivation

Interactive rendering• Rapid feedback is paramount

- High accuracy is less important

• Fast consistent framerate- Eg, > 5 fps

- Could use faster renderer• Eg, hardware accelerated scan conversion

• Use same renderer- Need to bridge framerate gap

Visual Feedback Loop

Standard visual feedback loop• Entirely

synchronous• Framerate is

limited by the renderer

renderer

application

image

user

Visual Feedback Loop

Modified visual feedback loop

display

application

image

userrenderer

Asynchronousinterface

Goals

Independent display process• Works with many (pixel-based) renders

Exploit frame to frame coherence• Reproject pixels from previous frames

Fast consistent framerate• Use simple, fast methods

Concentrate rendering effort• Prioritize pixel’s need for recomputation

Demo

Previous Work

Approximate or progressive approaches• Progressive radiosity or ray tracing• Frameless rendering

Reprojection or warping• Image based rendering (IBR)• Ray tracing acceleration

Previous Work

Parallel processing• Multiprocessors or distributed clusters

Intelligent display processes• Post-rendering warp• Holodeck system

Algorithm Overview

renderer

renderer

image

Rendercache

interpolate

sampling

depth cull

project

Display process

Image Estimation

Projection• Project cached points onto current image

- Camera transform provided by application

• Z-buffer- In case multiple points map to a pixel

Image Estimation

Problem: visual artifacts

Original view New view

Image Estimation

Depth cull heuristic• Problem: occluded points may be visible

- Z-buffering only works within a pixel

• Find pixels with locally inconsistent depths- Likely to be from different occluding surfaces

Raw projection After depth cull

Image Estimation

Interpolation / smoothing• Problem: small gaps in point data• Interpolate pixel colors

- Compute locally-weighted average colors

- Currently uses 3x3 neighborhoods

After interpolationAfter depth cullRaw projection

Image Estimation

Results after each stage

Projection Depth cull Interpolation

Image Estimation

Problem: visual artifacts• Simple filter can remove many artifacts

Need new points from renderer• Previously invisible areas• Color changes due to non-diffuse shading

- Eg,specular highlights

• Changes due to user editing- Changes in lighting, geometry, materials

Sampling

Generate priority image• Based on pixel’s need for re-rendering• Priority given to pixels with older points

- Render cache stores an age with each point

• Empty pixels priority based on local density- Highest priority given to regions without points

Sampling

Choose pixels for rendering• Sampling must be sparse

- Relatively few pixels are rendered per frame

• Chosen using error-diffusion dither- Concentrates pixels in high priority regions

- Maintains good spatial distribution

• Requested pixels sent to renderer(s)- Results returned at some later frame

Sampling

Displayed image Priority image Requested pixels

Optimizations

Further prioritizing sampling• Identify points that are likely to be outdated

- Color change heuristic

- Renderer supplied hints

• Prematurely age these points- Forces sooner resampling of these points

Optimizations

Moving objects• Application can supply object transforms• Applied to points in the render cache

- Improves tracking of moving objects

- Points also aged to encourage resampling

Demo

Results

Timing: 70.5 ms or 14 fps• 256x256 image, display process only• 195 Mhz R10K processor

Other(10%)

Point projection(47%)

Depth cull(12%)

Interpolation(20%)

Request new samples(8%)

Update render cache(4%)

Conclusions

Greatly improved interactivity• Eg, ray tracing, path tracing

Efficient reuse of rendered pixels• Using reprojection and simple filters

Prioritized sparse sampling• Efficently uses limited rendering budget

Independent automatic display process• Can be used with many different renderers

Future Work

Larger images• Cost scales linearly with # of pixels

Higher render ratios• Currently work well out to about 1:64

Anti-aliasing

The End

Overview

Cache rendered results• Stored as colored points in 3D

Estimate current image• Project points onto current image plane• Filter to reduce artifacts

Prioritize future rendering• Identify problem pixels• Sparse sampling for limited render budget