Differential Instant Radiosity for Mixed Reality

Martin Knecht, Christoph Traxler, Oliver Mattausch, Werner Purgathofer, Michael Wimmer

Institute of Computer Graphics and AlgorithmsVienna University of Technology

Motivation

2Martin Knecht

Motivation

3Martin Knecht

Problem statement

Virtual objects should seamlessly blend into the real scene

Provide a plausible illusion

Simulation of light interaction between real and virtual objects is necessary

Solution should run at interactive to real-time frame rates

4Martin Knecht

Related Work

Image Based LightingRitschel and Grosch used a HDR camera to capture environment

Hierarchical Importance Sampling, Clarberg

Real-time Global IlluminationInstant Radiosity introduced by Keller

Imperfect Shadow Maps from Ritschel et. al

Merging Real and Virtual ScenesDifferential Rendering introduced by Fournier and extended by Debevec

Differential Photon Mapping from Grosch5Martin Knecht

Assumptions

Geometry of real scene known

BRDFs of real surfaces are known

Surrounding illumination is distant

6Martin Knecht

Differential Instant Radiosity for MR

Idea: Combine Differential Rendering (DR) with Instant Radiosity (IR)

Why? DR adds virtual objects to real scene

IR fast method to compute GI solution in real-time

But: Two solutions are needed for DR!

not really

How? Keep track on the light paths

7Martin Knecht

GI - Instant Radiosity

Use Virtual Point Lights (VPLs) to approximate global illumination

8Martin Knecht

GI - Instant Radiosity

Use Virtual Point Lights (VPLs) to approximate global illumination

9Martin Knecht

GI - Instant Radiosity

Use Virtual Point Lights (VPLs) to approximate global illumination

10Martin Knecht

GI - Instant Radiosity

Use Virtual Point Lights (VPLs) to approximate global illumination

11Martin Knecht

GI - Instant Radiosity

Use Virtual Point Lights (VPLs) to approximate global illumination

12Martin Knecht

Differential Rendering

Method to add virtual objects into real scene images

Compute two global illumination solutions and add only difference to real scene image

+ Minimizes visual error from wrong BRDF approximations

- We need to calculate two GI solutions in real-time

13Martin Knecht

Differential Rendering

Difference BufferGrey means no difference

GI(Real + Virtual) GI(Real)

14Martin Knecht

Differential Rendering

Masked Video Image Difference Buffer

Final Image15Martin Knecht

How to get GI(Real+Virtual) and GI(Real)?Keep track on the light paths

L: Lightsource

D: Diffuse light bounce

S: Specular light bounce

E: Eye

Heckberts classification

16Martin Knecht

How to get GI(Real+Virtual) and GI(Real)?Keep track on the light paths

L: Lightsource

D: Diffuse light bounce

S: Specular light bounce

E: Eye

Direct illumination: LDE

Heckberts classification

17Martin Knecht

How to get GI(Real+Virtual) and GI(Real)?Keep track on the light paths

L: Lightsource

D: Diffuse light bounce

S: Specular light bounce

E: Eye

Direct illumination: LDE

1st bounce diffuse indirect illumination: LDDE

Heckberts classification

18Martin Knecht

Differential Instant Radiosity for MR

Calculate shading result only once

Decide on path information to which GI solution the result belongs

L[x]D[x]E

L[x]D[x]D[x]E

[x] can be „real“ or „virtual“

All „real“ then add to GI(Real+Virtual) and GI(Real)

One [x] „virtual“, only add to GI(Real+Virtual)19Martin Knecht

LRDRDRE path

Differential Instant Radiosity for MR

20Martin Knecht

GI(Real + Virtual) GI(Real)

LRDRDRE path

Added to both Buffers

Differential Instant Radiosity for MR

GI(Real + Virtual) GI(Real)

21Martin Knecht

LRDRDVE path

Differential Instant Radiosity for MR

22Martin Knecht

GI(Real + Virtual) GI(Real)

LRDRDVE path

Add only to the

GI(Real+Virtual)

Buffer

Differential Instant Radiosity for MR

GI(Real + Virtual) GI(Real)

23Martin Knecht

Differential Instant Radiosity for MR

Take blocking geometry into account

24Martin Knecht

LRDRDRE path

Differential Instant Radiosity for MR

25Martin Knecht

GI(Real + Virtual) GI(Real)

LRDRDRE path

Virtual red wall blocks light!

Differential Instant Radiosity for MR

GI(Real + Virtual) GI(Real)

26Martin Knecht

Limitations

27Martin Knecht

Limitations

Double shadowing

Incosistent color bleeding

28Martin Knecht

Results

Test deviceCore2 Quad CPU Q9550 at 2.8GHz

8GB RAM

NVIDIA Geforce GTX 285 with 1GB VRAM

Windows 7 64-Bit

C# with SlimDX to access DirectX 10

29Martin Knecht

Results

30Martin Knecht

Results

31Martin Knecht

Results

32Martin Knecht

Results

33Martin Knecht

Our method uses instant radiosity to simulate mutual influence between objects

Direct Illumination

Indirect Illumination

Combination of Differential Rendering and Instant Radiosity

Lightsources, objects, camera can be dynamic

Cannot take all paths into account

Works reasonable fast (20 – 30 fps)

Conclusion

34Martin Knecht

Consider all light paths

Add camera artifacts

Camera-based, real-time reconstruction of geometry

Real-time BRDF estimation

Future Work

35Martin Knecht

Thank you for your attention!

36Martin Knecht