Practical SVBRDF Acquisition of 3D Objects with...

Practical SVBRDF Acquisition of 3D Objects with Unstructured Flash Photography

KAIST Universidad de Zaragoza, I3A †

† † †§Giljoo Nam Joo Ho Lee Diego Gutierrez Min H. Kim

§

2/41

Input & Output

Unstructured flash photographs 3D geometry

3/41

Input & Output

Spatially-varying BRDFUnstructured flash photographs

4/41

Input & Output

Reproduction in a virtual environmentUnstructured flash photographs

5/41

Previous Work

[Holroyd et al., 2010] [Schwartz et al., 2013]

[Lensch et al., 2003] [Tunwattanapong et al., 2013]

6/41[Riviere et al., 2015]

[Ren et al., 2011]

Previous Work

[Aittala et al., 2015]

[Hui et al., 2017]

7/41

Challenges

• Limited sampling angles for BRDF acquisition

objectcamera &

light

How can we reconstruct a full BRDF from retro-reflected observations?

8/41

Challenges

• Simultaneous acquisition of SVBRDF and 3D geometry

objectcamera &

light

How can we ensure 3D-2D correspondence?

same point

9/41

BRDF Acquisition from Flash Photography

objectcamera &

light

n

o

df

h

hq

dq

i

( , ) ( , , )h d df f q q f=i o

• Parameter space for isotropic BRDFs

a 3D BRDF data

dq

hq

df

[Rusinkiewicz 98] [Matusik 02]

10/41

BRDF Acquisition from Flash Photography

a full 3D BRDF

dq

hq

df

subsample(flash photography)

hq

df

5dq » °object

camera &light

• Limited BRDF sampling angle in flash photography

11/41

• Cook-Torrance BRDF with 1D data-driven NDF

BRDF Model Validation


( ) ( )( )( ) ( , , ) ( , ),4

dsD G Ff r r

p= +

× ×h n i o h ii on i n o

reconstructed BRDF

dq

hq

df

subsample(flash photography)

hq

df

5dq = °

a full 3D BRDF

dq

hq

dfC-T BRDFw/ 1D NDF

fit BRDF

[MERL BRDF dataset]

12/41

BRDF Model Validation


( ) ( )( )( ) ( , , ) ( , ),4

dsD G Ff r r

p= +

× ×h n i o h ii on i n o

reference reconstructed difference0

1

13/41

SVBRDF Representation

• Basis BRDFs and spatial blending weights

object

( )3, ;f i o x

( )2, ;f i o x

( ) ( )1

, ; ,B

b bb

f fw=

=åi o x i o

( )1, ;f i o x

( )( ), ; :SVBRDF

, : basis BRDF

: blending weight: # of basis BRDFs

b

f

f

Bw

i o x

i o

14/41

Problem Definition

, , ,minimize

bX N W F

• Objective function using photometric consistency

-2

photographs rendering

X: 3D verticesN: surface normals

W: blending weightsFb: basis BRDFs

15/41

Problem Definition

( )2, , , 1 1

minimize ( ; ) ( , ; ) ( ; )( )P K

p kL f L

= =

- - ×ååbX N W F

o x i o x i x n i

• Objective function using photometric consistency

for all images & for all 3D vertices

captured radiance reconstructed radiance



16/41

Overview

• Iterative & alternating optimization of {X, N, W, Fb}

initialize update W update Fb update N update X

update SVBRDF



17/41

Data Capture

• Any camera with a flash will work

• We test two types of cameras

– Mobile phone with built-in LED

– DSLR with built-in flash

• 100 - 200 images (still/video)

example images

18/41

Initialization

• Initial geometry from image-based 3D modeling COLMAP [Schönberger 2016]: SfM è MVS è meshing

point cloud high-res mesh low-res mesh(our initial geo.)

point cloud high-res mesh low-res mesh

19/41

Initialization



initial geo.point clouds

20/41

Initialization



point cloud high-res mesh low-res mesh

21/41

Initialization


point cloud high-res mesh our result point cloud high-res mesh our result

22/41

Overview

• Iterative & alternating optimization of {X, N, W, Fb}



initialize update W update Fb update N update X

update SVBRDF

23/41

2

, , , ,, 1 1 1

minimizeP K B

p k p k p k p b bp k b

v f w= = =

æ ö¢ -ç ÷è ø

åå åbW F

Φ f

Update SVBRDF

captured reflectance reconstructed reflectance

• Re-formulate the equation w.r.t. the reflectance

( )2, , , 1 1

minimize ( ; ) ( , ; ) ( ; )( )P K

p kL f L

= =

- - ×ååbX N W F

o x i o x i x n i

update W update Fb update N update X

update SVBRDF


24/41

• Per-vertex optimization

Update SVBRDF: Blending Weights

2,

1

1minimize s.t. 0, 12

B

b p bb

w w=

- > =åωQω r


update SVBRDF

é ù é ùé ùê ú ê úê úê ú ê ú=ê úê ú ê úê úê ú ê úë û

ë û ë û

Q ω r

geometric factors& basis BRDFs

weights(unknowns)

observations

25/41

Update SVBRDF: Basis BRDFs

• Optimize for all observations (NOT per-vertex)

• Hold W fixed and solve for Fb• Sparse quadratic programming

2

, , , ,1 1 1

minimizeP K B

p k p k p k p b bp k b

v f w= = =

æ ö¢ -ç ÷è ø

åå åbF

Φ f

observation basis BRDFs(unknowns)

estimated weights

update W update Fb update N update Xupdate SVBRDF

26/41

Update Normals


• Non-linear optimization è hard to solve!

( )21

minimize ( ; ) ( , ; , ) ( ; )( )K

kL f L

=

- - ×ån

o x i o x n i x n i

normal for BRDF normal for irradiance


update SVBRDF

K: # of images

27/41

Update Normals


• Linear regression è easy to solve!

geometric normal( , known)

shading normal( , unknown)


update SVBRDF

( )21

minimize ( ; ) ( , ; , ) ( ; )( )K

kL f L

=

- - ×ån

o x i o x n i x n i!

K: # of images

n!n

( )21

minimize ( ; ) ( , ; , ) ( ; )( )K

kL f L

=

- - ×ån

o x i o x n i inx !

28/41

Update Geometry

• Screened Poisson surface reconstruction [Kazhdan 2015]

2 2minimize ( ) ( ) ( )p

p p p pdc

c a cÎ

-Ñ + åòx X

V x x x x

3D vector field fromshading normals

gradient of new geometry

squared distance ofcurrent and new geometry

: new geometry (unknown)c

update W update Fb update N

update SVBRDF

update X

29/41

Update Geometry

• Screened Poisson surface reconstruction [Kazhdan 2015]

shading normalinput

geometryestimated

shading normals Nupdate

geometry X

mesh vertex

mesh edge

GT surface

update W update Fb update N

update SVBRDF

update X

30/41

Iterative Optimization

• Iterate the whole process until RMSE of the test set starts to increase

geometric difference

0.00

0.01

0.02

0.03

1 3 5 7

Hau

sdor

ff di

stan

ce

iteration

photo consistency error

0.087

0.095

0.103

0.111

1 3 5 7

RM

SE

iteration

train test

update W update Fb update N update Xupdate SVBRDF

31/41

Results

32/41

Results

33/41

Results

34/41

Comparison with Previous Work

reference(NextEngine)

Xia SIGA'16(passive env. lighting)

Ours(active flashlight)

35/41

Evaluation: Geometry Refinement

high low ours

0.10

63

0.09

17

0.05

93

avg. geo. diff. [mm]

reference (NextEngine)

COLMAP (low res.)

oursCOLMAP (high res.)

36/41

Evaluation: Geometry Refinement

reference (NextEngine)

COLMAP (low res.)

oursCOLMAP (high res.)

high low ours

0.10

63

0.09

17

0.05

93

avg. geo. diff. [mm]

37/41

Discussion: DLSR vs. Mobile Phone

• DLSRs

– Higher sensor signal-to-noise ratio

– Brighter flashlight intensity

– Does not need a darkroom

experiment setup flash + indoor lightflash in a darkroom

SNR: 85.22 dB

38/41

Discussion: DLSR vs. Mobile Phone

• Mobile phones

– Better portability

– Record a video with the LED light on

39/41

Limitation

• Geometry– Complex geometries cannot be reconstructed accurately from

image-based 3D modeling.• Pinecones, hair strands, etc.

• Material Appearance– Inter-reflections

– Subsurface scattering– Transparency

40/41

Conclusion

• Capture images using commercial cameras with flash lights

• You will get high-quality 3D geometry & SVBRDF!

Practical SVBRDF Acquisition of 3D Objects with...

Documents

Transcript of Practical SVBRDF Acquisition of 3D Objects with...