1 Surface Reflectance Estimation and Natural Illumination Statistics Ron Dror, Ted Adelson, Alan...

1

Surface Reflectance Estimation and Natural Illumination Statistics

Ron Dror, Ted Adelson, Alan Willsky Artificial Intelligence Lab, Lab for Information and Decision

Systemshttp://www.ai.mit.edu/people/rondror

July 13, 2001

2

Reflectance Estimation Problem Surface appearance depends on surface

reflectance, illumination, and geometry. We wish to estimate reflectance under

unknown illumination.

3

Human vision

4

Machine vision

5

Motivation

Recognize materials. Reflectance, like texture, is a primary visual

characteristic of materials. Material recognition is important in its own

right and as a complement to shape recognition.

Capture real-world reflectances for rendering purposes.

Rectify classical motion, stereo, and shape-from-shading algorithms.

6

Reflectance estimation is ill-posed

A surface’s BRDF f(i,i; r,r) specifies how much of the light incident from any one direction is emitted in any second direction.

The brightness of a surface patch to a viewer is a weighted integral over illumination from all directions.

Goal: estimate reflectance (function of 4 variables) from an image (function of 2 variables) under unknown illumination from every direction (function of 2 variables at every point on the surface).

More degrees of freedom than measurements, even assuming known geometry, homogeneous reflectance.

7

Bayesian formulation Find the most likely reflectance given

image data. Given image data R, find most likely

reflectance f by marginalizing over illumination I.

P(f) – prior probability of a reflectance function P(I) – prior probability of an illumination field

Challenges: P(f) and P(I) are not readily available. Integration over all illuminations is

computationally daunting!

Iff

dIIfRPIPfPRfPf ),|()()(maxarg)|(maxargˆ

8

Two simplified formulations

1. Classification (finite but arbitrary classes):

2. Parameter estimation using a reflectance model (regression).

9

Prior information: illumination Assuming distant light sources, we can

represent illumination by a single spherical image.

Projection of spherical map

Rendered surfaces

10

Statistical models of illumination Illumination maps possess statistical

regularities akin to those of “natural images”. Histogram of pixel

intensities Histogram of wavelet coefficients

11

Importance of illumination statistics for humans People recognize reflectance more

easily under realistic illumination than simplified illumination.

12

Ward reflectance model A physically realizable variant of the Phong

model (satisfies energy conservation and reciprocity).

d: proportion of incident radiation reflected diffusely.

s: proportion of incident radiation reflected specularly.

: surface roughness, or blur in specular component.

2

22

4

)/tanexp(

coscos

1),;,(

ri

sd

rriif

diffuse component

specular lobe

13

Effect of Ward model parameters on pixel intensity histogram

Original

pixel intensity

pro

babili

ty

14


Original d=.1

pixel intensity

pro

babili

ty

15


Original d=.2

pixel intensity

pro

babili

ty

16


Original d=.3

pixel intensity

pro

babili

ty

17


Original d=.4

pixel intensity

pro

babili

ty

18


Original d=.4 s=.1

pixel intensity

pro

babili

ty

19


Original d=.4 s=.2

pixel intensity

pro

babili

ty

20


Original d=.4 s=.3

pixel intensity

pro

babili

ty

21


Original d=.4 s=.4

pixel intensity

pro

babili

ty

22


Original d=.4 s=.4 =0

pixel intensity

pro

babili

ty

23


Original d=.4 s=.4 =.05

pixel intensity

pro

babili

ty

24



pixel intensity

pro

babili

ty

25



pixel intensity

pro

babili

ty

Dependence of statistics on reflectance parameters

Real-world illuminations

Random checkerboard illuminations

d s

d s

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

norm

aliz

ed

deri

vati

ve

norm

aliz

ed

deri

vati

ve

27

Each reflectance clusters in feature space

black matte black shiny

white matte white shiny

gray shiny chrome

28

A system for classification

“Learn” relationships between features of the observed image and reflectance classes.

For a distant viewer and convex object, radiance depends only on local surface orientation.

29

Implementation flow chart

This leaves two open questions: How to select relevant statistics? How to build a classifier?

30

SVM classifier Support vector machines are relatively robust

to the inclusion of extraneous features. A sample classifier based on just two statistics:

black matte black shiny

white matte white shiny

gray shiny chrome

31

Training data sets

6 Ward model reflectances, 9 illuminations (Debevec)

11 Ward model reflectances, 100 illuminations (Teller)

9 real spheres, photographed at seven locations

32

Performance

Rendered:6 BRDFs, 9 illums

Rendered:11 BRDFs, 100 illums

Photos:9 spheres,

7 illums

Chance 16.7% 9.1% 11.1%

6 hand-selected features

98.1% 98.5% 93.7%

6 auto-selected features

96.3% 94.4% 74.6%

6 PCA features

79.6% 86.8% 71.4%

33

Conclusions

Our classifier rivals human performance when geometry is known and reflectance is homogeneous.

Although ill-posed, reflectance estimation under unknown natural illumination is tractable.

The statistical structure of natural illumination plays an essential role in visual reflectance estimation by humans and machines.

34

Future directions In progress or submitted:

Extension to complex or unknown geometry; robustness to incorrect assumed geometry.

Quantitative study of natural illumination statistics. Measurement of human ability to estimate reflectance from a

single image without contextual information. Additional goals:

Rigorous theoretical foundation – link illumination statistics directly to selected features.

Estimate spatially varying reflectance.

35

Misclassifications

Illumination Misclassified image

Potential source of confusion

36

Feature selection By hand, based on insights

developed through work with Ward model.

Using automated feature selection method, which iterates the following steps:

Estimate marginal probability density of each feature for each class.

Select the feature that minimizes Bayes error.

Regress remaining features against selected features, and subtract off predicted values.

37

Auto-selected features 6 features selected based on images of spheres

with 6 Ward model reflectances under 9 illuminations:

10th percentile of 4th finest vertical subband 90th percentile of pixel intensity variance of 3rd finest diagonal subband 10th percentile of pixel intensity 90th percentile of 4th finest vertical subband median of 3rd finest horizontal subband

Hand-selected features mean and 10th percentile of original image variance of two finest vertical subbands ratio of these two variances kurtosis of second finest vertical subband

38

Complex vs. simple illumination People recognize reflectance more easily under

realistic illumination than simplified illumination.

A reflectance estimation algorithm which takes advantage of natural illumination statistics will fail for atypical illumination.

39

Human reflectance estimation Pool balls

Note ambiguity in overall color and brightness when matte spheres are viewed in isolation.

40

Related Work Yu, Debevec, Malik, and Hawkins, ’99

Reflectance and illumination from multiple photos. Sato, Wheeler, and Ikeuchi, ’97

Reflectance and geometry from photos and laser range finder, with known illumination.

Marschner, Greenberg, et al., ’98, ’99 Reflectance under known illumination.

Tominaga and Tanaka, 1999, ’00 Reflectance and geometry under simple lighting,

using color separation. Pellacini, Ferwerda, Greenberg, ’00

Perceptually uniform gloss space for graphics. Ramamoorthi and Hanrahan, ’01

Determine when reflectance estimation problem is well-posed.

41

Photographic data Nine different spheres under the same

illumination.

42

Photographic data Same spheres under a second illumination.

43

Photographic data Same spheres under a third illumination.

44

Illumination conditions

45

Rendered data set

6 spheres under one illumination condition

46

Rendered data set

6 spheres under a 2nd illumination condition

47

Is this task even possible? Humans are good at it.

In psychophysical tests, we found that humans could match synthetic images of surfaces with similar reflectances rendered under different real-world illuminations.

Two conclusions: Humans rely on prior information in estimating reflectance. Humans estimate reflectance without explicitly estimating

illumination.

Photographs of three

spheres under two

illumination conditions

48

Additional applications Rectify motion, stereo, and shape-from-

shading algorithms.

Capture real-world reflectances for rendering purposes.

Yu, Debevec, Malik, Hawkins, SIGGRAPH 1999

Gideon Stein, unpublished

49

Debevec spheres

Dependence of statistics on reflectance parameters

Real-world illuminations

Random checkerboard illuminations

d s

d s

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

norm

aliz

ed

deri

vati

ve

norm

aliz

ed

deri

vati

ve

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

kurt

mean

skew

10

% 5

0%

90

%

v

ar

1 Surface Reflectance Estimation and Natural Illumination Statistics Ron Dror, Ted Adelson, Alan...

Documents

Transcript of 1 Surface Reflectance Estimation and Natural Illumination Statistics Ron Dror, Ted Adelson, Alan...