Sebastian Enrique Columbia University [email protected] Relighting Framework COMS 6160 –...

Sebastian EnriqueColumbia University

[email protected]

Relighting Framework

COMS 6160 – Real-Time High Quality Rendering

Nov 3rd, 2004

COMS 6160 Relighting Framework

Sebastian Enrique - Columbia University - [email protected]

Nov 3rd, 2004

2

About Relighting

Why it is useful?

Photorealistic real-time rendering of complex scenes with complex illumination is an open problem. One IBR approach is to capture or render a set of original images from a scene, and then relight it in real time to produce the same scene but with novel illumination!

IMAGES

+ =

LIGHTSRELIGHTED

SCENE

What is Relighting?

“Given a set of different illuminated images from a scene, relighting is the process of producing new images of the same scene with new lighting conditions, composing in some way the original data.”

Sebastian Enrique - Columbia University - [email protected] 3

About Relighting (cont.)

What is one of the most challenging parts of it?

High quantity of images is needed to produce good results, the challenge is to find/use a good compression technique in order to relight fast and using as less memory as possible.

IMAGES

?

COMPRESSEDIMAGES

COMS 6160 Nov 3rd, 2004Relighting Framework


Some Previous Work

• Doorsey et al. used a progressive radiosity method where they pre-rendered synthetic scenes to simulate lighting conditions –superimposing single light source images- in opera stages in 1991.

• Many (Hallinan ‘94, Epstein ‘95, etc.) have pre-acquired real images changing the lighting direction.

• Debevec compressed the pre-acquired images in JPG and processed in the compressed domain in 2000.

• Sloan uses low-frequency spherical harmonics on geometry in 2002.

• In 2003 Sloan uses clustered or VQPCA on spherical harmonic coefficients.

• Ng compressed data using wavelets in 2003.

• Sloan (Local Deformable PRT) and Ramamoorthi (Triple Product Wavelets) are the most recent related techniques, 2004.



LSD Relighting Approach

• The Lighting Sensitive Display (LSD) was developed by Shree Nayar, Peter Belhumeur, and Terry Boult.

• Basically, it is a monitor with an attached camera that captures the lighting conditions of the environment. The monitor shows an scene, which changes (is relighted) as the illumination in the environment changes.

• They adopted an image-based approach, using a large set of images.



LSD Relighting Approach (cont.)

• They have developed a novel algorithm (using two stages of Principal Component Analysis or PCA) that compresses that large set of images and allows the relighting in real-time with complex lighting conditions.

• They have reached compression ratios of 476:1 for colored images.

• The algorithm simultaneously exploits correlations over the lighting domain as well as coherences over the spatial domain of the image.



LSD Relighting Approach (cont.)

• We based our Relighting Framework on the LSD approach.

• In the following slides we will explain:

• How the input images for the LSD algorithm should be taken.• First compression stage of the algorithm.• Second compression stage of the algorithm.• Real-Time rendering.

• Then, we will get on:

• Extending the LSD approach with the use of cubemaps.• Problems found.• Future directions.• Discussion and Videos



Setting Up Images

Fixed Viewpoint

Grid of Positions for DistantLight Sources, it’s in the frontface of an imaginary cube.

Distant Light SourceFor Single Image

Scene to Render / Capture

• To take the images, a grid of light source positions is generated in the front face of an imaginary cube (a plane parallel to the image plane).

• For each position on the grid, place the distant light with maximum intensity and generate (render or capture) an image.



First Stage SVD

• The first SVD (Singular Value Decomposition) exploits the fact that locally (within small regions of the images) the variation due to changes in illumination can be approximated by a small number of bases.

• The image is divided into m square blocks, each containing p pixels, and bases are computed for each.

• Each image Ii is an image of the scene illuminated by a single distant point light source.

IMAGES

Image divided in msquare blocks

Each block has p pixels

Image Ii



First Stage SVD (cont.)

• Iij denote the jth block of the ith image.

• For each block in the scene a low-dimensional approximation is computed as follows:

• Ij is a p x n matrix representing a collection of image blocks. The I column of Ij is formed by p pixels of the jth block.

• Using SVD, a rank b approximation to Ij is found as:

• Ej is a p x b column-orthogonal matrix, called block bases.• Sj is a b x b diagonal matrix.• Cj is an n x b column-orthogonal matrix.

• The singular values from Sj are absorbed into CjT, getting a b x n matrix, Lj, called block lighting coefficients.

Tjjjj CSEI

jjj LEI

Each Ij represents the pixels for the same block on all images


Ij Sj

CjT

p x n p x n

n x n

= x x

diagonal matrix

(singular values)



Ij

CjT

p x n

b x b b x n

x x

Ej

p x bSj

• Applying Rank b:

• SVD:

Ej

p x p

Ij

Lj

p x n

b x n

x

Ej

p x b

• Abosorving Sj values into CiT:




• Then, stacking all of the m Lj block matrices, we get the lighting coefficient matrix L.

• The image bases m Ej are also stacked in the matrix E.

• The collection of submatrices within E and L contain all the information needed to approximate the collection of images corresponding to the n lighting directions.

L(m * b) x n

E(m * p) x b




Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10



Second Stage SVD

• The fact that there is much coherence in image blocks is exploited using a second SVD.

• It is applied to the light coefficient matrix L:

• U is an (m x b) x q column-orthogonal matrix, called lighting coefficient bases.• V is a q x n matrix, called compressed coefficient matrix.• Rank q denotes the number of linear bases kept to approximate L.

UVL

L

V

(m * b) x n

q x n

x

U(m * b) x q



Second Stage SVD (cont.)

Image: 640x480; n=64x64=4096; m=40x30=1200; p=16x16=256; b=10; q=200



Examples

Relighted ImageSVD First Stage Rank 3

Original Image

Bad Rank Election(showing only red color component)

Note block discontinuities!



Examples (cont.)

Relighted ImageSVD First Stage: Rank 10

SVD Second Stage: Rank 20

Original Image(in fact, two combined original images)

Higher RanksOnly small differences in

brightness are noticeable, butthere are no visible block

discontinuities.



Real-Time Rendering

• Real-Time illumination is represented by the illumination field vector s.

]1,0[...21 in sssss

]1,0[...21 iT

n sssss

• Each element si corresponds to a point light, and its value represents the intensity that that point light is contributing with to the scene.

• To render the relighted scene in real-time, preprocessed matrices E, U, and V should be used:

)(VsUI

1. Compute a compressed coefficient vector as the product (q elements)

2. Compute a lighting coefficient vector I as (m*b elements)

3. To render the j block, a subvector Ij from I must be multiplied with the corresponding stacked Ej matrix. This must be done each frame for the m blocks.

Vs

)...( 2)1(1)1( bjjbjbj IIII jjj IER (b elements) (p elements)



Real-Time Rendering (cont.)

]1,0[...21 in sssss



Demo: Point Lights / David

Image: 480x640; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20



Extending LSD: Cube Lighting

• In place of using only the front face of the cube to capture images with different lighting condition, use all of the 6 faces.

• The parameters that change are the quantity of input images and the length of the s light field vector (both now 6 * n).

• In this way, the scene can be relighted with illumination coming for all around.

Grid of Positions for DistantLight Sources in the Full Cube

Distant Light SourceFor Single Image

Fixed Viewpoint



High-Dynamic Range Cubic Environment Map

• Having the relighting scheme for the full cube, we can add cubic environment maps to relight the scene.

• To have more accuracy, High-Dynamic Range (HDR) cubic environment maps are used (floating point values in place of 0..255).

• The CubicMap could be rotated, and each element in the s light field vector is contributed by the value of the corresponding texel, weighted by the solid angle.

3222

2322

)(

14.4)cos()(

)()()(

zyx rrrmR

RN

mr

dAiw

iwiHDRis

• HDR(i) is the corresponding texel value; N is the corresponding cube face normal; R is the vector from the origin to the center of the element i in the cube grid; m is the face cube grid resolution (squared is how many images per face exist); each cube face has a size of 2 x 2.



Demo: CubeMap & Point Lights / Nicole

Image: 512x512; n=6x16x16=1536; m=32x32=1024; p=16x16=256; b=10; q=20



Framework Summary

PREPROCESS(SVD 1 & 2)

IMAGES

Eb = […]Ub = […]Vb = […]

MATRICES

Eb = […]Ub = […]Vb = […]

MATRICESREAL-TIME

RELIGHTINGFRAMEWORK

RELIGHTEDSCENE

HDR

Generate Scene-Matrices For Images Set Of Each Scene to Relight

Relight in Real-Time the Given Scene-Matrices with HDR CubeMap and Point Lights



Problems Found

• Preprocess takes TOO MUCH TIME.

• Preprocess uses TOO MUCH MEMORY.

• Few things could be done in current graphics hardware (HDR cubemap processing specially).



Future Directions

• Optimize preprocessing stages (distributed computations?)

• Use error metrics to automatically select adequate ranks.

• Extend user interface to allow the relighting using lower ranks than that given in the input matrices.

• Allow viewpoint changes.



End of Talk

Ready to hear…

• Comments

• Suggestions

• Discussions

• Questions

More VIDEOS to show while chatting…


Sebastian Enrique Columbia University [email protected] Relighting Framework COMS 6160 –...

Documents

Transcript of Sebastian Enrique Columbia University [email protected] Relighting Framework COMS 6160 –...