What the spider’s eyes don’t tell the spider’s...
Transcript of What the spider’s eyes don’t tell the spider’s...
![Page 1: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/1.jpg)
What the spider’s eyes don’t tell the spider’s brain
![Page 2: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/2.jpg)
Depth Perception from Image Defocus in a Jumping Spider
(*) “Depth Perception from Image Defocus in a Jumping Spider” Nagata, Koyanagi, Tsukamoto, Saeki, Isono, Shichida, Tolunaga, Kinoshita, Arikawa, and Terakita.
![Page 3: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/3.jpg)
How to judge distance to prey?
• World is 3-D — images are 2-D
Can’t determine distance monocularly
Scale factor ambiguity
![Page 4: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/4.jpg)
How to judge distance to prey?
• World is 3-D — images are 2-D
Can’t determine distance monocularly
Scale factor ambiguity
• Many “depth cues”
Ratio image size to object size
Ratio image motion to object motion
...
![Page 5: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/5.jpg)
How to judge distance to prey?
• World is 3-D — images are 2-D
Can’t determine distance monocularly
Scale factor ambiguity
• Many “depth cues”
Ratio image size to object size
Ratio image motion to object motion
...
• Lens accomodation
![Page 6: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/6.jpg)
How to judge distance to prey?
• World is 3-D — images are 2-D
Can’t determine distance monocularly
Scale factor ambiguity
• Many “depth cues”
Ratio image size to object size
Ratio image motion to object motion
...
• Lens accomodation
• Binocular stereo
![Page 7: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/7.jpg)
How to judge distance to prey?
• World is 3-D — images are 2-D
Can’t determine distance monocularly
Scale factor ambiguity
• Many “depth cues”
Ratio image size to object size
Ratio image motion to object motion
...
• Lens accomodation
• Binocular stereo
• Defocus blur
![Page 8: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/8.jpg)
Accomodation? (1/f = 1/a+ 1/b)
![Page 9: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/9.jpg)
Accomodation? (1/f = 1/a+ 1/b)
![Page 10: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/10.jpg)
Accomodation? (1/f = 1/a+ 1/b)
![Page 11: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/11.jpg)
Binocular stereo?
(*) “Jumping Spider Vision”, David Hill, Wikipedia
![Page 12: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/12.jpg)
Defocus blurring?
![Page 13: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/13.jpg)
Defocus blurring?
PSF
R
![Page 14: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/14.jpg)
Defocus blurring? 2J1(Rρ)/(Rρ)
PSF
R
MTF
3.8317 / R
![Page 15: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/15.jpg)
Multi-layer retina
(*) “Depth Perception from Image Defocus in a Jumping Spider” Nagata, Koyanagi, Tsukamoto, Saeki, Isono, Shichida, Tolunaga, Kinoshita, Arikawa, and Terakita.
![Page 16: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/16.jpg)
Depth from two image planes
(*) “Depth Perception from Image Defocus in a Jumping Spider” Nagata, Koyanagi, Tsukamoto, Saeki, Isono, Shichida, Tolunaga, Kinoshita, Arikawa, and Terakita.
![Page 17: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/17.jpg)
What’s wrong with that model?
![Page 18: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/18.jpg)
What is wrong with that model?
• Assumes image on back layer (L1) is always in focus
But this would require accomodation;
then there is no need for anything else!
![Page 19: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/19.jpg)
What is wrong with that model?
• Assumes image on back layer (L1) is always in focus
But this would require accomodation;
then there is no need for anything else!
• Assumes blur on front layer (L2) depends on distance
If back is in focus then the blur in front is fixed;
blur in front merely reflects inter image layer spacing!
![Page 20: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/20.jpg)
What is wrong with that model?
• Assumes image on back layer (L1) is always in focus
But this would require accomodation;
then there is no need for anything else!
• Assumes blur on front layer (L2) depends on distance
If back is in focus then the blur in front is fixed;
blur in front merely reflects inter image layer spacing!
• Assumes amount of blur can be ascertained from image
Problem is ill posed; for example:
Blurry image of sharp texture same assharp image of blurry texture!
![Page 21: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/21.jpg)
Some possible approaches
• Transport of Intensity Equation (TIE)
∇xy ·(I(x, y, z)
∇xyφ(x, y, z)
k
)= −∂I(x, y, z)
∂z
![Page 22: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/22.jpg)
Some possible approaches
• Transport of Intensity Equation (TIE)
∇xy ·(I(x, y, z)
∇xyφ(x, y, z)
k
)= −∂I(x, y, z)
∂z
• Light-field propagation
![Page 23: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/23.jpg)
Some possible approaches
• Transport of Intensity Equation (TIE)
∇xy ·(I(x, y, z)
∇xyφ(x, y, z)
k
)= −∂I(x, y, z)
∂z
• Light-field propagation
• Deconvolution
• . . .
![Page 24: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/24.jpg)
System model
E(x,y)
⊗ b1(x,y)
⊗ b2(x,y)
E1(x,y)
E2(x,y)
![Page 25: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/25.jpg)
Solution based on this
E(x,y)
⊗ b1(x,y)
⊗ b2(x,y)
E1(x,y)
E2(x,y)
⊗ b2(x,y)
⊗ b1(x,y)
−
![Page 26: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/26.jpg)
Solution based on this
E(x,y)
⊗ b1(x,y)
⊗ b2(x,y)
E1(x,y)
E2(x,y)
⊗ b2(x,y)
⊗ b1(x,y)
− b1 ⊗ b2 = b2 ⊗ b1
![Page 27: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/27.jpg)
Doing it in parallelE(x,y)
⊗ b(z) ⊗ b(z+d)
E1(x,y) E2(x,y)
⊗ b(1+d) ⊗ b(1)−
mag
⊗ b(2+d) ⊗ b(2)−
mag
⊗ b(3+d) ⊗ b(3)−
mag
⊗ b(4+d) ⊗ b(4)−
mag
argmin
![Page 28: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/28.jpg)
0.00 mm
![Page 29: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/29.jpg)
0.15 mm
![Page 30: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/30.jpg)
0.30 mm
![Page 31: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/31.jpg)
0.45 mm
![Page 32: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/32.jpg)
0.60 mm
![Page 33: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/33.jpg)
0.75 mm
![Page 34: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/34.jpg)
0.90 mm
![Page 35: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/35.jpg)
1.05 mm
![Page 36: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/36.jpg)
1.20 mm
![Page 37: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/37.jpg)
Recovery of in-focus distance
0 1 2 3 4 5 6
0
1
2
3
4
![Page 38: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/38.jpg)
Recovering the “in focus” image
• Ill-posed problem from single defocused image:
P1(u) = P(u)M1(u)
• can’t recover frequency components where M1(u) = 0.
![Page 39: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/39.jpg)
Recovering the “in focus” image
• Ill-posed problem from single defocused image:
P1(u) = P(u)M1(u)
• can’t recover frequency components where M1(u) = 0.
• But with two images — defocused to different degrees:
P2(u) = P(u)M2(u)
P1(u)M∗1 (u)+ P2(u)M∗
2 (u) = P(u)(‖M1(u)‖2 + ‖M2(u)‖2
)• works as long as, for any u, either M1(u) �= 0 or M2(u) �= 0.
• (actually, use Wiener filtering)
![Page 40: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/40.jpg)
What the spider’s eyes don’t tell the spider’s brain
![Page 41: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/41.jpg)
Pillbox convolved with pillbox is not a pillbox
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0
![Page 42: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/42.jpg)
Calibration of lens motion
60
61
62
63
0 10 20
f (mm)
steps60
61
62
63
0 10 20
f (mm)
steps
![Page 43: What the spider’s eyes don’t tell the spider’s brainpeople.csail.mit.edu/bkph/talkfiles/Jumping_Spiders_Old.pdf · 2017. 6. 16. · What the spider’s eyes don’t tell the](https://reader035.fdocuments.us/reader035/viewer/2022071415/6111870ae27dc32f5e5ec826/html5/thumbnails/43.jpg)
Lens motion from estimates of zeros in DFT
1208 1209 1210 1211 1212 1213 1214 1215 12160.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
1.1
1.2
1.3
mm
frame