A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi...
-
date post
20-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi...
A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering
A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering
Ravi [email protected]
Ravi [email protected]
OutlineOutlineOutlineOutline
• Motivation• Forward Rendering• Inverse Rendering• Object Recognition
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation• Forward Rendering• Inverse Rendering• Object Recognition
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
Interactive RenderingInteractive RenderingInteractive RenderingInteractive Rendering
Directional Source Complex Illumination
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
Reflection MapsReflection MapsReflection MapsReflection Maps
Blinn and Newell, 1976
Environment MapsEnvironment MapsEnvironment MapsEnvironment Maps
Miller and Hoffman, 1984Later, Greene 86, Cabral et al. 87
Reflectance Space ShadingReflectance Space ShadingReflectance Space ShadingReflectance Space Shading
Cabral, Olano, Nemec 1999
Reflectance MapsReflectance MapsReflectance MapsReflectance Maps
• Reflectance Maps (Index by N)• Horn, 1977
• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984
• Reflectance Maps (Index by N)• Horn, 1977
• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984
Mirror Sphere Chrome Sphere Matte (Lambertian) Sphere
Irradiance Environment Map
Complex IlluminationComplex Illumination Complex IlluminationComplex Illumination
• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)
• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)
• New representation for lighting design, IBR
• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)
• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)
• New representation for lighting design, IBR
Directional Source Complex LightingIllumination Irradiance
Environment Map
Photorealistic RenderingPhotorealistic RenderingPhotorealistic RenderingPhotorealistic Rendering
Geometry
70s, 80s: Splines 90s: Range Data
Materials/Lighting(Texture, reflectance [BRDF], Lighting)
Realistic Input Models Required
Arnold Renderer: Marcos Fajardo
Rendering Algorithm
80s,90s: Physically-based
Inverse RenderingInverse RenderingInverse RenderingInverse Rendering• How to measure realistic material models, lighting?
• From real photographs by inverse rendering
• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data
• How to measure realistic material models, lighting?• From real photographs by inverse rendering
• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data
Illumination:Mirror Sphere
Grace Cathedralcourtesy
Paul Debevec
BRDF (reflectance): Images using
point light source
FlowchartFlowchartFlowchartFlowchart
Lighting
BRDF NewView
NewView,Light
ResultsResultsResultsResults
Photograph Computer rendering
New view, new lighting
Ramamoorthi and Hanrahan, SIGGRAPH 2001a
Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals
• Complex (possibly unknown) illumination
• Estimate both lighting and reflectance (factorization)
• Complex (possibly unknown) illumination
• Estimate both lighting and reflectance (factorization)
Photographs of 4 spheres in 3 different
lighting conditions
courtesy Dror and Adelson
Factorization AmbiguitiesFactorization AmbiguitiesFactorization AmbiguitiesFactorization Ambiguities
Width of Light Source
SurfaceRoughness
Inverse ProblemsInverse ProblemsInverse ProblemsInverse Problems
• Sometimes ill-posed• No solution or several solutions given data
• Often numerically ill-conditioned• Answer not robust, sensitive to noise
• Need general framework to address these issues• Mathematical theory for complex illumination
• Sometimes ill-posed• No solution or several solutions given data
• Often numerically ill-conditioned• Answer not robust, sensitive to noise
• Need general framework to address these issues• Mathematical theory for complex illumination
Directional
Source
Area source
Same reflectance
Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
Peter Belhumeur: Yale Face Database A
OutlineOutlineOutlineOutline
• Motivation
• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)
2
2
ˆ( , ) ( , ) ( , )o i i o iB L d
( , ) ( ) ( )i i iL L L
o
L
i i
o B
BRDF
( , )i o
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
B L
Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
iIl
pleL ,ˆ i oIl I
lp
qp
p
e e ,oIp
pIl
ql
p
B e e
, ,ˆ2l p l l pB L
Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)
-1-2 0 1 2
0
1
2
.
.
.
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
Spherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic Analysis
, ,ˆ2l p l l pB L
lL ,ˆl p,l pB
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
2D:
3D:,lm pqB
lmL ,ˆlq pqIsotropic
, ,ˆlm pq l lm lq pqB L
InsightsInsightsInsightsInsights
• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
• Inverse rendering is deconvolution
• Our contribution: Formal Frequency-space analysis
• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
• Inverse rendering is deconvolution
• Our contribution: Formal Frequency-space analysis
Example: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDF
• BRDF is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to lighting
• Mirror Sphere (Gazing Ball)
• BRDF is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to lighting
• Mirror Sphere (Gazing Ball)
Phong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet Models
• Rough surfaces blur highlight
• Analytic Formula• Approximately Gaussian
• Rough surfaces blur highlight
• Analytic Formula• Approximately Gaussian
Mirror Matte
2
exp2l l
l
s
Roughness
Example: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDF
2 1
2
2
2 1 ( 1) !ˆ 2
4 ( 2)( 1) 2 !
l
l l l
l ll even
l l
Ramamoorthi and Hanrahan, JOSA 2001
Second-Order ApproximationSecond-Order ApproximationSecond-Order ApproximationSecond-Order Approximation
Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial
Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial
L=0 L=1 L=2 Exact
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x ySimilar to Basri & Jacobs 01
Dual RepresentationDual Representation Dual RepresentationDual Representation
• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation
• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation
Frequency9 param.
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Angular SpaceFrequency9 param.
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
VideoVideoVideoVideo
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
Peter Belhumeur: Yale Face Database A
Face Basis FunctionsFace Basis FunctionsFace Basis FunctionsFace Basis Functions
• 5 basis functions capture 95% of image variability
• Linear combinations of spherical harmonics
• Complex illumination not much harder than points
• 5 basis functions capture 95% of image variability
• Linear combinations of spherical harmonics
• Complex illumination not much harder than points
Frontal Lighting Side Above/Below Extreme Side Corner
Inverse LightingInverse LightingInverse LightingInverse Lighting
• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:
contradicts theorem in Preisendorfer 76
• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters
• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:
contradicts theorem in Preisendorfer 76
• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters
B L
BL
Inverse LambertianInverse LambertianInverse LambertianInverse LambertianSum l=2 Sum l=4True Lighting
Mirror
Teflon
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals• Formal study: Well-posedness, conditioning
• General Complex (Unknown) Illumination
• Formal study: Well-posedness, conditioning
• General Complex (Unknown) Illumination
Bronze Delrin Paint Rough Steel
Photographs
Renderings(Recovered
BRDF)
Quantitative Pixel error approximately 5%
Factoring the Light FieldFactoring the Light FieldFactoring the Light FieldFactoring the Light Field
• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)
(½ 3D) -- Make use of reciprocity
• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)
(½ 3D) -- Make use of reciprocity
,,
1 lm pqlq pq
l lm
B
L
,00
00, 0
1 lmlm
l l
BL
B
00
1
l
L
B L
Algorithms ValidationAlgorithms ValidationAlgorithms ValidationAlgorithms ValidationRendering
Unknown
Lighting
Photograph
Known
Lighting
Kd 0.91 0.89 0.87
Ks 0.09 0.11 0.13
1.85 1.78 1.48
0.13 0.12 .14
Recovered
Light
Estimate by ratio of
intensity andtotal energy
Marschner
Complex GeometryComplex GeometryComplex GeometryComplex Geometry
3 photographs of a sculptureComplex unknown illuminationGeometry KNOWNEstimate BRDF and Lighting
FlowchartFlowchartFlowchartFlowchart
Lighting
BRDF NewView
NewView,Light
ComparisonComparisonComparisonComparison
RenderedKnown Lighting
Photograph RenderedUnknown Lighting
New View, LightingNew View, LightingNew View, LightingNew View, Lighting
Photograph Computer rendering
Textured ObjectsTextured ObjectsTextured ObjectsTextured Objects
Real RenderingComplex, Known Lighting
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary • Conclusions• Pointers
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary • Conclusions• Pointers
SummarySummarySummarySummary
• Reflection as Convolution
• Signal-Processing Framework
• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter
• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization
• Efficient Forward Rendering (Environment Maps)
• Lighting Variability in Object Recognition
• Reflection as Convolution
• Signal-Processing Framework
• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter
• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization
• Efficient Forward Rendering (Environment Maps)
• Lighting Variability in Object Recognition
PapersPapersPapersPapers
• http://graphics.stanford.edu/~ravir/research.html
• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]
• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]
• http://graphics.stanford.edu/~ravir/research.html
• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]
• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]
AcknowledgementsAcknowledgementsAcknowledgementsAcknowledgements
• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• John Parissenti
• Jean Gleason
• Scanned cat sculpture is “Serenity” by Sue Dawes
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant #0085864: “Interacting with the Visual World”
• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• John Parissenti
• Jean Gleason
• Scanned cat sculpture is “Serenity” by Sue Dawes
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant #0085864: “Interacting with the Visual World”
The EndThe EndThe EndThe End
Related WorkRelated WorkRelated WorkRelated Work
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Related WorkRelated WorkRelated WorkRelated Work
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case
Example: Directional SourceExample: Directional SourceExample: Directional SourceExample: Directional Source
• Lighting is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter
• Lighting is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Incomplete sparse data: Few views• Use practical Dual Representation
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Angular SpaceFrequency
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Concavities: Self Shadowing
• Incomplete sparse data: Few views• Use practical Dual Representation
• Concavities: Self Shadowing
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Reflected RayShadowed?
IntegrateLighting
SourceShadowed?
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Concavities: Self Shadowing
• Textures: Spatially Varying Reflectance
• Incomplete sparse data: Few views• Use practical Dual Representation
• Concavities: Self Shadowing
• Textures: Spatially Varying Reflectance
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Reflected RayShadowed?
IntegrateLighting
SourceShadowed?
Kd(x) Ks(x)
Inverse BRDFInverse BRDFInverse BRDFInverse BRDF
• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting
• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting
Directional
Source
Area source
Same BRDF
B L
B
L
ComparisonComparisonComparisonComparisonRenderingPhotograph
Kd 0.91 0.89
Ks 0.09 0.11
1.85 1.78
0.13 0.12
Marschner Our method
KnownLighting
Pixel ErrorApproximately 5%