Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction
description
Transcript of Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction
![Page 1: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/1.jpg)
Mahdi M. Bagher / Cyril Soler / Kartic Subr / Laurent Belcour / Nicolas HolzschuchMahdi M. Bagher / Cyril Soler / Kartic Subr / Laurent Belcour / Nicolas Holzschuch
Interactive rendering of Interactive rendering of acquired materials on dynamic geometryacquired materials on dynamic geometry
using bandwidth predictionusing bandwidth prediction
![Page 2: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/2.jpg)
• Direct reflection under distant illumination with no visibility and global illumination effects
• Rendering an image is computationally expensive.
Shading
fro
m W
ikip
edi
a
Illumination BRDF
![Page 3: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/3.jpg)
Acquired Materials
• Using measured materials• photo-realistic rendering• large memory footprint
(33MB for a single isotropic BRDF)
• Not suitable for real-time rendering
Phong shading Measured “Color-Changing-Paint-3”BRDF measurement Gantry [Matusik2002]
![Page 4: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/4.jpg)
Monte Carlo Sampling
• Monte Carlo approximation to the shading integral
• Importance sampling• less noisy
• Data driven reflectance• no analytical importance function
• pre-computed Importance samples
Monte Carlo sampling
BRDF importance sampling
![Page 5: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/5.jpg)
Sampling
• Dealing with two types of sampling• Reconstruction• sampling in image space• to render an image
• Integration• sampling in the space of incident directions • to shade each individual pixel
Illumination BRDF Cosine
![Page 6: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/6.jpg)
The Problem
• Shading is slow…
• How to speed-up shading?
![Page 7: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/7.jpg)
Case Study
• What difference various materials make in rendering an image?
Measured reflectance data from [Matusik2002]
![Page 8: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/8.jpg)
Diffuse Materials
• Diffuse materials• shading varies slowly across the image (low frequencies)• many integration samples for each pixel (wide BRDF lobe)
Measured Teflon [Matusik2002] BRDF Lobe [BRDFLab]
![Page 9: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/9.jpg)
Specular Materials
• Specular materials• shading varies quickly across the image (high frequencies) • few integration samples for each pixel (narrow BRDF lobe)
Measured color-Changing-Paint3 [Matusik2002] BRDF Lobe [BRDFLab]
![Page 10: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/10.jpg)
The Problem
• Rendering is slow…• How to speed-up shading?
• Can we exploit the coherency between • Reconstruction samples…• Integration samples…
• …for adaptive sampling?
• How to measure the coherency between samples?
![Page 11: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/11.jpg)
Intuition
• Study the frequencies in the image• As a function of what happens to the light• Traveling from the light source to the camera
![Page 12: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/12.jpg)
The Traveling Light
• Light when traveling is affected by• Travel through free space• Object’s curvature• Reflectance• BRDF• Texture
• Occlusion
![Page 13: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/13.jpg)
Our Contribution
• How to speed-up shading?By sparse sampling the image• How to combine the sparse samples?• Multi-resolution rendering and bi-lateral up-sampling
By adaptive sampling for integration
![Page 14: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/14.jpg)
Our Contribution (Illustrated)
Sparse reconstruction samples Adaptive Integration samples
,
Final shaded image
Multi-resolution shading and bi-lateral up-sampling
∝ the average variance of the shading integrand∝ the local maximum variations in the image
![Page 15: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/15.jpg)
Assumptions
• Direct reflection under distant illumination
• No visibility and global illumination effects
• Allow dynamic editable geometry
• Screen-space rendering in the context of deferred-shading
• Any shading model is possible
• Save more if shading cost is higher e.g. measured materials
• BRDF importance sampling
Normal
![Page 16: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/16.jpg)
Dynamic Geometry Animation
QuickTime™ and a decompressor
are needed to see this picture.
![Page 17: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/17.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 18: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/18.jpg)
Related Work GPU Rendering of Complex Materials
[Heidrich&Seidel1999]
[Ramamoorthi2002]
[Kautz2002]
[Claustres2007]
[Kautz&McCool1999]
[Latta&Kolb2002]
[McCool2001]
[Wang2009]
![Page 19: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/19.jpg)
Related WorkMulti-Resolution Screen-Space Algorithms
[Nichols&Wyman2009] [Shopf2009][Nichols&Wyman2010]
[Soler2010]
[Nichols2010][Segovia2006]
![Page 20: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/20.jpg)
Related WorkPre-Computed Transport
[Sloan2002]
[Ramamoorthi2009]
[Sun2007]
[Liu2004]
[Loos2011]
[Loos2012]
![Page 21: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/21.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 22: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/22.jpg)
Goal
• Sparse sampling for reconstruction• ∝ maximum local variations (local bandwidth)
• Adaptive sampling for integration• ∝ variance of the shading integrand
• Multi-resolution rendering and bilateral up-sampling.• a single coarse-to-fine rendering pass
![Page 23: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/23.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Local light-field definition
• Spectral Operations
• 2D local bandwidth
• Image-space local bandwidth
• Adaptive sampling for integration
• Validation
• …
![Page 24: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/24.jpg)
Local Light-Field
• 4D local light-field illustrated in 2D
Central ray
Sp
ace
Angle
![Page 25: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/25.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Local light-field definition
• Spectral Operations
• 2D local bandwidth
• Image-space local bandwidth
• Adaptive sampling for integration
• Validation
• …
![Page 26: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/26.jpg)
Spectral Operations
•Spectral operations [Durand et al. 2005][D
urand
et al. 2005]
![Page 27: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/27.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Local light-field definition
• Spectral Operations
• 2D local bandwidth
• Image-space local bandwidth
• Adaptive sampling for integration
• Validation
• …
![Page 28: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/28.jpg)
2D Local Bandwidth
• 2D Local Bandwidth• maximum local variations about a light path both in angle and space.
spatial angular
2D local bandwidth
![Page 29: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/29.jpg)
Propagating Local Bandwidth
![Page 30: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/30.jpg)
Propagating Local Bandwidth
• One-bounce 2D bandwidth propagation• Transport (from light to reflector)• Reflection (BRDF & Texture)• Transport (from reflector to pixel)
• Matrix operators on 2D bandwidth
where
Bandwidthat pixel p
Reflectionoperator
Illumination bandwidth
Scale Curvature Mirror BRDF and texture
Curvature ScaleTransport to image plane
Transport to the surface
![Page 31: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/31.jpg)
Input Signals
• Illumination• environment map
• Reflectance• Acquired isotropic BRDFs
• TexturesGold-paint[ Matusik2003]
Grace cathedral [Debevec2001]
![Page 32: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/32.jpg)
Input Signals’ Bandwidth
• Illumination• environment map• Purely angular bandwidth
• Reflectance• Acquired isotropic BRDFs• Purely angular bandwidth
• Textures• Purely spatial bandwidth
Gold-paint[ Matusik2003]
Grace cathedral [Debevec2001]
![Page 33: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/33.jpg)
Computing Local Bandwidth
• Discrete Wavelet Transform to compute bandwidth• environment map and reflectance (BRDF & Texture)
• Discrete Wavelet Transform instead of WFFT • wavelets are well localized both in space and frequency.
Space Frequency
Daubechies 4 tap wavelet
FFT
![Page 34: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/34.jpg)
Local Bandwidth Examples
Gold-paint[ Matusik2003]
Angular bandwidth
Angular bandwidthGrace cathedral [Debevec2001]
![Page 35: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/35.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Local light-field definition
• Spectral Operations
• 2D local bandwidth
• Image-space local bandwidth
• Adaptive sampling for integration
• Validation
• …
![Page 36: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/36.jpg)
Image-Space Bandwidth
• We combine bandwidth from various incident directions to get the final bandwidth at a given pixel.
• Taking a max is too conservative, we take the weighted average instead.
• Bandwidth at each pixel
Bandwidth estimate based on max
Bandwidth estimate Rendered image
![Page 37: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/37.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 38: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/38.jpg)
Variance of the Shading Integrand
• # shading samples ∝ sum of bandwidths, weighted by the illumination times reflectance squared.
• The sum is a conservative approximation of the actual variance of the shading integrand.
Angular bandwidth arriving at the surface
Angular bandwidth of the reflectance
![Page 39: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/39.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 40: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/40.jpg)
Validation of the Bandwidth and Variance
Estimated variance Measured varianceEstimated bandwidth Measured bandwidth(WFFT)
![Page 41: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/41.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 42: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/42.jpg)
The Algorithm
• Step 1• Compute g-buffers• Load Illumination and reflectance bandwidth
• Step 2• Estimate bandwidth and variance
• Step 3• Mip-map bandwidth and variance
• Step 4• Shade and up-sample
![Page 43: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/43.jpg)
Video: algorithm
QuickTime™ and a decompressor
are needed to see this picture.
![Page 44: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/44.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 45: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/45.jpg)
Result (Final Image)
![Page 46: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/46.jpg)
Result (Bandwidth)
![Page 47: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/47.jpg)
Result (Variance)
![Page 48: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/48.jpg)
Timings
• Fast bandwidth/variance estimation
• (8 ms)
(ms)(ms)
![Page 49: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/49.jpg)
Timings
• Computation time scales linearly with the total number of shading samples
![Page 50: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/50.jpg)
Comparison with Reference
Our algorithm(1015 ms)
Similar time(906 ms)
Similar quality(2639 ms)
![Page 51: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/51.jpg)
Comparison with Spherical Gaussian Approximation (Wang et al. 2009)
Fast but different from ground-truth (notice that color changing effects are missing)
Path-traced reference
Interactive but more accurate
![Page 52: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/52.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 53: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/53.jpg)
ExtensionAdaptive Multi-Sample Anti-Aliasing
![Page 54: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/54.jpg)
Video: adaptive MSAA
QuickTime™ and a decompressor
are needed to see this picture.
![Page 55: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/55.jpg)
Timings
• Sub-linear performance for MSAA
![Page 56: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/56.jpg)
Roadmap
• Related work
• Our technique
• The goal
• Sparse sampling for reconstruction
• Adaptive sampling for integration
• Validation
• The rendering algorithm
• Results and comparisons
• Extension: MSAA
• Conclusion / Future work
![Page 57: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/57.jpg)
Conclusion
• Interactively shading dynamic geometry with acquired materials• Shade a fraction of pixels using bandwidth• Adaptively sample shading integrals• Combine shaded pixels using up-sampling
• Exploiting bandwidth to sub-linearly scale MSAA with deferred shading
![Page 58: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/58.jpg)
Future Work
• Local light sources
• Better up-sampling algorithm
• Depth-of-field
• 6D bandwidth computation for SVBRDFs
• Visibility and indirect illumination
![Page 59: Interactive rendering of acquired materials on dynamic geometry using bandwidth prediction](https://reader035.fdocuments.us/reader035/viewer/2022062314/568149e3550346895db70afd/html5/thumbnails/59.jpg)
Thank you
Interactive rendering of Interactive rendering of acquired materials on dynamic geometryacquired materials on dynamic geometry
using bandwidth predictionusing bandwidth prediction