1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological...
-
Upload
madeline-harrison -
Category
Documents
-
view
216 -
download
2
Transcript of 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological...
![Page 1: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/1.jpg)
1
Orientation fields and 3D shape Orientation fields and 3D shape estimationestimationRoland W. FlemingMax Planck Institute
for Biological Cybernetics
![Page 2: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/2.jpg)
2
Cues to 3D ShapeCues to 3D Shape
specularities shading texture
Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)
![Page 3: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/3.jpg)
3
specularities shading texture
Goal: Find commonalities between cues.
Cues to 3D ShapeCues to 3D Shape
![Page 4: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/4.jpg)
4
Cues to 3D ShapeCues to 3D Shape
![Page 5: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/5.jpg)
5
Cues to 3D ShapeCues to 3D Shape
Fleming, Torralba, Adelson
Todd and colleagues
Mingolla and Grossberg
Koenderink and van Doorn
Zucker and colleagues
Zaidi and Li
Malik and Rosenholtz
![Page 6: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/6.jpg)
6
It is remarkable that we can recover 3D shape:
No motion No stereo No shading No texture
image consists of nothing more than a distorted reflection of the world surrounding the object
Ideal mirrored surface
Fleming et al. (2004). JOV
Shape from SpecularitiesShape from Specularities
![Page 7: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/7.jpg)
7
As the object moves from scene to scene, the image changes dramatically.
Yet, somehow we are able to recover the 3D shape.
Shape from SpecularitiesShape from Specularities
![Page 8: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/8.jpg)
8
Curvatures determine distortionsCurvatures determine distortions
highly curved
![Page 9: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/9.jpg)
9
Curvatures determine distortionsCurvatures determine distortions
slightlycurved
Anisotropies in surface curvature lead to powerful distortions of the reflected world
![Page 10: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/10.jpg)
10
Interpreting distorted reflectionsInterpreting distorted reflections
![Page 11: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/11.jpg)
11
Orientation fieldsOrientation fields
Ground truth
![Page 12: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/12.jpg)
12
3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface
![Page 13: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/13.jpg)
13
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 14: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/14.jpg)
14
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 15: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/15.jpg)
15
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 16: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/16.jpg)
16
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 17: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/17.jpg)
17
ShinyShiny
Painted Painted
![Page 18: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/18.jpg)
18
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 19: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/19.jpg)
19
Latent orientationLatent orientationstructurestructure
![Page 20: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/20.jpg)
20
Orientation fieldsOrientation fieldsin shadingin shading
![Page 21: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/21.jpg)
21
Orientation fieldsOrientation fieldsin shadingin shading
![Page 22: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/22.jpg)
22
Reflectance as IlluminationReflectance as Illumination
a(f) = 1 / f
= 0 = 0.4 = 0.8 = 1.2
= 1.6 = 2.0 = 4.0 = 8.0
![Page 23: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/23.jpg)
23
highly curved
![Page 24: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/24.jpg)
24
slightlycurved
Anisotropies in surface curvature lead to anisotropies in the image.
![Page 25: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/25.jpg)
25
Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic
materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
![Page 26: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/26.jpg)
26
Idea: Experiment 1Idea: Experiment 1
Rationale: measure stability of 3D shape across changes in surface reflectance
Method: gauge figure task? Problem: costly to do full depth reconstruction for
many shapes and materials Solution? Compare sparse gauge measurement? Alternative task?:
locate depth extrema along given raster line (2D task)
![Page 27: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/27.jpg)
27
TextureTexture
Anisotropic compression of texture depends on surface slant
![Page 28: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/28.jpg)
28
TextureTexture
Anisotropic compression of texture depends on surface slant
![Page 29: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/29.jpg)
29
Orientation fieldsOrientation fieldsin texturein texture
![Page 30: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/30.jpg)
30
Orientation fieldsOrientation fieldsin texturein texture
![Page 31: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/31.jpg)
31
Orientation fieldsOrientation fieldsin texturein texture
![Page 32: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/32.jpg)
32
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
![Page 33: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/33.jpg)
33
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 34: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/34.jpg)
34
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 35: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/35.jpg)
35
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 36: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/36.jpg)
36
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 37: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/37.jpg)
37
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 38: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/38.jpg)
38
Idea: Experiment 2Idea: Experiment 2
Rationale: use orientation fields to predict misperceptions of 3D shape
Possible methods Gauge figure task?
Matching task: subject adjusts shear of a textured
object until it appears to match the shaded version of the same object
Subject adjusts shear of one oject (shaded or textured) until it appears to match the ‘degree of shear’ of another object? Sounds too strange?
![Page 39: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/39.jpg)
39
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
![Page 40: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/40.jpg)
40
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
![Page 41: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/41.jpg)
41
Idea: Experiment 3Idea: Experiment 3
Rationale: use orientation fields to predict misperceptions of 3D shape
Possible methods gauge figure task to reconstruct
full 3D shape. Again, this is costly, but perhaps
a few shapes are enough
depth extrema task: locate depth extrema along raster line (this is what Todd and Thaler did). Potentially we could predict the
locus directly from the orientation field
![Page 42: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/42.jpg)
42
Idea: Experiment 3Idea: Experiment 3 Compare small and large changes in orientation field by using texture stretching along
the line of sight Advantage: same infringement of ‘isotropy assumption’, different change in apparent
3D shape
UnstretchedUnstretchedStretched 2:1Stretched 2:1
along line of sightalong line of sight
![Page 43: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/43.jpg)
43
Uses biologically plausible measurements
Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).
Potential of Potential of Orientation FieldsOrientation Fields
![Page 44: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/44.jpg)
44
No need for visual system to estimate reflectance or illumination explicitly.
Classical shape from shading uses the reflectance map to estimate surface normals from image intensities
Reflectance map is usually unknown and ambiguous
Potential of Potential of Orientation FieldsOrientation Fields
![Page 45: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/45.jpg)
45
Stable across albedo discontinuities.
Breton and Zucker (1996), Huggins and Zucker (2001)
Potential of Potential of Orientation FieldsOrientation Fields
![Page 46: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/46.jpg)
46
Handle improbable combinations of reflectance and illumination.
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
Potential of Potential of Orientation FieldsOrientation Fields
![Page 47: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/47.jpg)
47
We could measures shape estimates with these types of stimuli as well.
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
Link back toLink back toexperiment 1 ?experiment 1 ?
![Page 48: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/48.jpg)
48
May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts
Potential of Potential of Orientation FieldsOrientation Fields
Ohad Ben-Shahar
![Page 49: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/49.jpg)
49
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
( )2
Latent shadingLatent shading
![Page 50: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/50.jpg)
50
( )2
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
Latent shadingLatent shading
![Page 51: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/51.jpg)
51
ConclusionsConclusions
Orientation fields are potentially a very powerful source of information about 3D shape
For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought
![Page 52: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/52.jpg)
52
Thank youThank youCollaborators
Ted AdelsonAntonio Torralba
Funding
RF supported byDFG FL 624/1-1
![Page 53: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/53.jpg)
53
What still needsWhat still needsto be explained?to be explained?
For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.
Therefore shading orientation fields vary considerably with changes in illumination.
sidefront top
![Page 54: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/54.jpg)
54
What still needsWhat still needsto be explained?to be explained?
Surprising prediction: 3D shape should actually be less stable across changes in illumination for diffuse than for specular materials.
We found evidence for changes in 3D shape with changes in illumination Alternative: higher order invariants establish an equivalence between
different orientation fields. Example: joint measures of orientation at different locations.
sidefront top
![Page 55: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/55.jpg)
55
Note analogy to textures of different orientations
Todd et al. (2004)
What still needsWhat still needsto be explained?to be explained?
![Page 56: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/56.jpg)
56
Matte dark grey
Rough metal
Glossy light grey
![Page 57: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/57.jpg)
57
PlasticsPlastics
(a) Mirror (b) Smooth plastic (c) Rough plastic
![Page 58: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/58.jpg)
58
When the world is anisotropicWhen the world is anisotropic
Brushed horizontally Brushed vertically
![Page 59: 1 Orientation fields and 3D shape estimation Roland W. Fleming Max Planck Institute for Biological Cybernetics.](https://reader036.fdocuments.us/reader036/viewer/2022062618/5514d6df55034640138b6377/html5/thumbnails/59.jpg)
59
Stability across changesStability across changesin surface reflectancein surface reflectance A parametric space of glossy plastic
materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s