Visualization- Determining Depth From Stereo
description
Transcript of Visualization- Determining Depth From Stereo
![Page 1: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/1.jpg)
Visualization-Determining Depth From Stereo
Saurav Basu
BITS Pilani
2002
![Page 2: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/2.jpg)
Introduction
Example of Human Vision Perception of Depth from Left and right eye
images Difference in relative position of object in
left and right eyes. Depth information in the 2 views??
![Page 3: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/3.jpg)
Basis for Stereo Imaging
LEFT EYE RIGHT EYE
LEFT VIEW RIGHT VIEW
![Page 4: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/4.jpg)
The Stereo Problem
– The stereo problem is usually broken in to two subproblems• Extraction of Depth information from Stereo
Pairs• Use of depth data to visualize the world
scene in 3-dimensions by a suitable projection technique.
![Page 5: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/5.jpg)
Stereo Images
Depth Estimation
Visualization
![Page 6: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/6.jpg)
What are Stereo Images?
Images of the same world scene taken from slightly displaced view points are called stereo images.To illustrate how a typical stereo imaging system let us take a look at the camera model for obtaining stereo images
![Page 7: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/7.jpg)
Camera Model Of A Stereo System
Image 2
Image 1
W (X,Y,Z)
BaseLine distance
y
x
x
y
(x2,y2)
(x1,y1)
Optical Axis
![Page 8: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/8.jpg)
Important Points about the Model
The cameras are identical The coordinate systems of both cameras are
perfectly alligned. Once camera and world co-ordinate systems are
alligned the xy plane of the image is alligned with the XY plane of the world co-ordinate system,then Z coordinate of W is same for both camera coordinate systems.
![Page 9: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/9.jpg)
Relating Depth with Image Coordinates
Z
W (X,Y,Z)
X
Origin Of
World Coordinate
System
B
Image 2
Image 1
(x1,y1)
(x2,y2)
Z -
![Page 10: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/10.jpg)
By Similar Triangles:
Disparity x2- x1where
Disparity
1
KB Depth, - Z
)(
)(
)(
)(
)(
21
21
22
1
11
1
21
12
22
2
11
1
Depthxx
KDepth
Putxx
BZ
ZxBX
ZxX
ZZZ
BXX
ZxX
ZxX
![Page 11: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/11.jpg)
Thus Depth is inversely proportional to (x1-x2) where x1 and x2 are pixel coordinates of the same world point when projected on the stereo image planes.
(x1- x2) is called the DISPARITY The problem of finding x1 and x2 in the stereo
pairs is done by a stereo matching technique.
Important Result
![Page 12: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/12.jpg)
Stereo Matching
– The goal of stereo matching algorithms is to find matching locations in the left and right images .
– Specifically find the coordinates of the pixel on the left and right images which correspond to the same world point.
– It is also called the correspondence problem.
![Page 13: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/13.jpg)
Correlation based approaches
– A common approach to finding correspondences is to search for local regions that appear similar
– Try to match a window of pixels on the left image with a corresponding sized window on the right image.
![Page 14: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/14.jpg)
Matching PixelsMatching Window
Diagram to illustrate the Stereo Matching
Assumption :Matching Pixels lie on same horizontal Raster Line(Rectified stereo)
Disparity of this pixel is 1 since x1=0 and x2=1,x2-x1=1
Left Image Right Image
![Page 15: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/15.jpg)
The SSD and SAD are commonly used correlation functions
2)),(),(()),(),,(( ydxIyxIydxIyxI rlrl
),(),()),(),,(( ydxIyxIydxIyxI rlrl
SSD:Sum of Squared Deviations
SAD:Sum of Absolute Differences
![Page 16: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/16.jpg)
The Multi Window Algorithm
In this algorithm technique 9 different windows are used for calculating disparity of a single pixel.
The window which gives the maximum correlation is used for disparity calculations.
![Page 17: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/17.jpg)
Left Image Right Image
Demonstration of the 9 different windows used for the Correlation
![Page 18: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/18.jpg)
Disparity Map
Based on the calculated disparities a disparity map is obtained
The disparity map is a gray scale map where the intensity represents depth.
The lighter shades (greater disparities) represent regions with less depth as opposed to the darker regions which are further away from us.
![Page 19: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/19.jpg)
Visualization
Visualization is the process by which I use the depth estimates from the stereo matching to build projections .
3-D information can be represented in many ways :
-Orthographic projections-Perspective Projections
![Page 20: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/20.jpg)
Perspective Projections
Perspective projections allow a more realistic visualization of a world scene
The visual effect of perspective projections is similar to the human visual system and photographic systems.
Hence perspective projection of the 3-d data was implemented for the stereo pairs.
![Page 21: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/21.jpg)
Center Of Projection
A
BA’
B’ Projection Plane
•In Perspective projections the projectors are of finite length and converge at a point called the center of projection.
•In perspective projection size of an object is inversely proportional to its distance from ooint of projection
Projectors
![Page 22: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/22.jpg)
Specifying a 3-D View
To specify a 3-d view we need to specify a projection plane and a center of Projection:
The Projection plane specified by 1. A point on the plane called the
View Reference Point (VRP)2. The normal to the view plane,i.e. View Plane Normal (VPN)
![Page 23: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/23.jpg)
We define a VRC (View Reference Coordinate system) on the projection plane with u,v,and n being its 3 axes forming a right handed coordinate system
The origin of the VRC system is the VRP The VPN defines the ‘n’ axis of the VRC system A View Up Vector (VUP) determines the ‘v’ axis of
the VRC system.The projection of the VUP parallel to the view plane is coincident with the ‘v’ axis.
![Page 24: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/24.jpg)
The ‘u’ axis direction is defined such that the ‘u’,’v’ and ‘n’ form a right handed coordinate system. A view Window on the view plane is defined ,projections lying outside the view window are not displayed .The coordinates (Umin,Vmin) and (Umax,Vmax) define this window. The center of projection Projection reference point(PRP).
![Page 25: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/25.jpg)
V
U
(Umin,Vmin)
(Umax,Vmax)
VPN
VRP
VIEW PLANE
Center of Projection
THE 3-D VIEWING REFERENCE COORDINATE SYSTEM
VUP
Window
N
CW
DOP
![Page 26: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/26.jpg)
The semi infinite pyramid formed by the PRP and the projectors passing through the corners of the view window form a view volume.
A Canonical view volume is one where the VRC system is alligned with the World Coordinate system.
![Page 27: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/27.jpg)
X or YBack Plane
Front Plane
-1
-1
1
Canonical view volume for Perspective Projections
-Z
The 6 bounding planes of the canonical view volume have equations:
x=z ,x=-z ,y=z, y=-z z=zmin, z=-1
PRP
![Page 28: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/28.jpg)
Perspective projection when VRC alligned with World Coordinate system
Y
Z
X
V
U
N
P(Xp,Yp,d)
P(X,Y,Z)
dPRP
CW
![Page 29: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/29.jpg)
1] /
/
[] z [1
matrix a
as drepresente becan tion Transforma The
d Z
dZY
Y
dZX
X
TrianglesSimilar From
.
0d100
0100
0010
0001
pp
ddz
y
dz
x
d
zyxz
yx
M
M
per
per
![Page 30: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/30.jpg)
Only true when view volume is canonical For arbitrary view volume -
First transform the view volume intocanonical form and then apply the above formula to take projections
For transforming a view volume we do the following: 1)Translate VRP to origin
2)Rotate VRC to allign u,v and n axes with the X,Y and Z axes.
![Page 31: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/31.jpg)
3)Translate the PRP to origin
4)Shear to make center line of view volume the the z-axis.
5)Scale such that the view volume becomes the canonical perspective view volume
![Page 32: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/32.jpg)
1. The translation matrix is
)(
1000
100
010
001
),,(
VRPT
dz
dy
dx
dzdydxT
2. The Rotation matrix is
1000
0
0
0
333
222
111
zyx
zyx
zyx
rrr
rrr
rrr
R
N
V
U
Z
X
YVRP
![Page 33: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/33.jpg)
)( 3.
][
][
][
2 2 2
1 1 1
3 3 3
PRPT
RRrrrR
RVUP
RVUPrrrR
VPN
VPNrrrR
where
xzzyxy
z
zzyxx
zyxz
![Page 34: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/34.jpg)
4. The Shear Matrix
dopz
dopyshy
dopz
dopxshx
Let
SH shy
shx
per
WindowofCenter :CW
Projection ofDirection :DOP
PRP-CWDOP
1000
0100
010
001
PRP
Y
-Z
CW
![Page 35: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/35.jpg)
5.The scale transformation
BVRPs
BVRPvv
VRPs
BVRPuu
VRPs
PRPTSHVRPLet
S
zz
z
zy
z
zx
per
per
'
1
)')((
'2
)')((
'2
1] 0 0 0).[(.'
1000
0s00
00s0
000s
minmax
minmax
z
y
x
PRP
Y
-Z
CW
Y=-Z
Y=-Z
![Page 36: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/36.jpg)
Once all the projected points have been calculated, scale the coordinates to fit the display screen.
A wire frame display of the image is obtained by joining the projections of all points lying on the same row or column.
Map the pixel colors of the image on to the projected points to create a realistic effect.
![Page 37: Visualization- Determining Depth From Stereo](https://reader035.fdocuments.us/reader035/viewer/2022062409/56814d5c550346895dba99f6/html5/thumbnails/37.jpg)
Limitations
Can work well only for stereo images whereminute details are not required.
More suited for depth estimation of landscape through images taken from top.
No accurate metric calculations done.