Computer Vision – Transformations, Imaging Geometry and Stereo...
Transcript of Computer Vision – Transformations, Imaging Geometry and Stereo...
![Page 1: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/1.jpg)
Computer Vision –Transformations,
Imaging Geometryand
Stereo Vision
Dr. S. DasIIT Madras, Chennai-36
![Page 2: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/2.jpg)
BASICSRepresentation of Points in the 3D world: a vector of length 3
[ ]TxX zy =
Right handed coordinate system
z
x
y
P(x,y,z) P’(x’,y’,z’)
T
Transformations of points in 3D
4 basic transformations
• Translation
• Rotation
• Scaling
• Shear
Affine transformations
![Page 3: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/3.jpg)
Basics 3D Transformation equations• Translation : P’ = P + ∆P
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
∆∆∆
+⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
zyx
zyx
zyx
'
'
'
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
z
y
x
SS
SS
000000
α
β
γ
• Scaling: P’= SP
• Rotation : about an axis,P’ = RP
![Page 4: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/4.jpg)
Positive Rotations: counter clockwise aboutthe origin
ROTATION - 2D
In matrix form, this is :
For rotations, |R| = 1 and [R]T = [R]-1. Rotation matrices are orthogonal.
⎥⎦
⎤⎢⎣
⎡=
)cos()sin()sin(-)cos(
θθθθ
R 0
5
4
3
2
1
1 2 3 4 5 X
Y
θ = 30°
θ
( ) ( )( ) ( )θθ
θθ cos sin' sin cos'
yxyyxx
+=−=
![Page 5: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/5.jpg)
• Translate by (-Px, -Py)
• Rotate
• Translate by (Px, Py)
Rotation about an arbitrarypoint P in space
As we mentioned before, rotations are applied about the origin. So to rotate about any arbitrary point P in space, translate so that P coincides with the origin, then rotate, then translate back. Steps are:
![Page 6: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/6.jpg)
P1
House at P1
Translation of P1 to Origin
Rotation by θ
θ
Translation back to P1
P1
Rotation about an arbitrarypoint P in space
![Page 7: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/7.jpg)
2D Transformation equations (revisited)• Translation : P’ = P + ∆P
⎥⎦
⎤⎢⎣
⎡∆∆
+⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡yx
yx
yx
''
• Rotation : about an axis,P’ = RP ⎥
⎦
⎤⎢⎣
⎡=
)cos()sin()sin(-)cos(
θθθθ
R
??1001
''
⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡∆∆
=⎥⎦
⎤⎢⎣
⎡yx
yx
yx T
⎥⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡ −=⎥
⎦
⎤⎢⎣
⎡''
cossinsincos
""
yx
yx
θθθθ
![Page 8: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/8.jpg)
Rgen = T1(-Px, -Py) * R2(θ) * T3(Px, Py)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−−
=100
1001
*1000)cos()sin(0)sin()cos(
*100
1001
y
x
y
x
PP
PP
θθθθ
Rotation about an arbitrarypoint P in space
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡+−−−−
=100
)sin(*)1)(cos(*)cos()sin()(sin(*)1)(cos(*)sin()cos(
θθθθθθθθ
xy
yx
PPPP
Using Homogeneous system
![Page 9: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/9.jpg)
Homogeneous representation of a point in 3D space:
point) 3D afor 1,w(|w z yx | T
==P
Transformations will thus be represented by 4x4 matrices:
P’ = A.P
![Page 10: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/10.jpg)
Homogenous Coordinate systems• In order to Apply a sequence of transformations to
produce composite transformations we introduce the fourth coordinate
• Homogeneous representation of 3D point: |x y z h|T (h=1 for a 3D point, dummy coordinate)
• Transformations will be represented by 4x4 matrices.
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∆∆∆
=
1000100010001
zyx
T
Homogenous Translation matrix
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
1 0 0 0000
00
0000
z
y
x
SS
S
S
Homogenous Scaling matrix
![Page 11: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/11.jpg)
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡−
=
1 0 0 0000
cossin0sincos0001
αααα
αR
Rotation about x axis by angle α
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡ −
=
1 0 0 0 000
1000cossin0sincos
γγγγ
γR
Rotation about z axis by angle γ
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−=
1 0 0 0 000
cos0sin
010sin0cos
ββ
ββ
βR
Rotation about y axis by angle β
Change of sign?
How can one do a Rotation about an arbitrary Axis in Space?
![Page 12: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/12.jpg)
3D Transformation equations (3)Rotation About an Arbitrary Axis in Space
Assume we want to perform a rotation about an axis in space, passing through the point (x0, y0, z0)with direction cosines (cx, cy, cz), by θ degrees.
1) First of all, translate by: - (x0, y0, z0) = |T|.2) Next, we rotate the axis into one of the principle
axes. Let's pick, Z (|Rx|, |Ry|).3) We rotate next by θ degrees in Z ( |Rz(θ)|).4) Then we undo the rotations to align the axis. 5) We undo the translation: translate by (x0, y0, z0)
The tricky part is (2) above.
This is going to take 2 rotations, i) about x (to place the axis in the x-z plane) and ii) about y (to place the result coincident with the z axis).
![Page 13: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/13.jpg)
Rotation about x by α: How do we determine α?
Project the unit vector, along OP, into the y-z plane. The y and z components are cy and cz, the directions cosines of the unit vector along the arbitrary axis. It can be seen from the diagram above, that :
α
xy
z
0P cz
cx
cy
d
d = sqrt(cy2 + cz
2), cos(α) = cz /d sin(α) = cy /d
xy
z
0
P(cx, 0 ,d)
d
cxβ
Rotation by β about y: How do we determine β? Similar to above:
![Page 14: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/14.jpg)
Determine the angle β to rotate the result into the Z axis: The x component is cx and the z component is d.
cos(β) = d = d /(length of the unit vector) sin(β) = cx = cx /(length of the unit vector).
Final Transformation: M = |T|-1 |Rx|-1 |Ry|-1 |Rz| |Ry| |Rx| |T|
If you are given 2 points instead, you can calculate the direction cosines as follows:
V = | (x1 -x0) (y1 -y0) (z1 -z0) |T
cx = (x1 -x0)/ |V| cy = (y1 -y0)/ |V| cz = (z1 -z0)/ |V|,
where |V| is the length of the vector V.
![Page 15: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/15.jpg)
Inverse transformations
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
∆−∆−∆−
=−
1000100010001
1
zyx
T
Inverse Translation
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=−
1000010000100001
1
z
y
x
SS
S
S
Inverse scaling
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡ −
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−1 0 0 0 000
1000cossin0sincos
1 0 0 0 000
cos0sin
010sin0cos
1 0 0 0000
cossin0sincos0
001γγγγ
ββ
ββ
αααα
Rα-1 Rγ
-1Rβ-1
Inverse Rotation
![Page 16: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/16.jpg)
Concatenation of transformations• The 4 X 4 representation is used to perform a
sequence of transformations. • Thus application of several transformations
in a particular sequence can be presented by a single transformation matrix
TSAvTvSRv ..RA ;))(( θθ ===∗
• The order of application is important… the multiplication may not be commutable.
![Page 17: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/17.jpg)
Cases where T1 * T2 = T2 * T1:
Commutivity of Transformations If we scale, then translate to the origin,
and then translate back, is that equivalent to translate to origin, scale, translate back?
When is the order of matrix multiplication unimportant?
When does T1 * T2 = T2 * T1?
rotationScale (uniform)rotationrotation
scalescaletranslationtranslation
T2T1
![Page 18: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/18.jpg)
COMPOSITE TRANSFORMATIONSIf we want to apply a series of
transformations T1, T2, T3 to a set of points, We can do it in two ways:
1) We can calculate p'=T1*p, p''= T2*p', p'''=T3*p''
2) Calculate T= T1*T2*T3, then p'''= T*p.
Method 2, saves large number of additions and multiplications (computational time) –needs approximately 1/3 of as many operations. Therefore, we concatenate or compose the matrices into one final transformation matrix, and then apply that to the points.
![Page 19: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/19.jpg)
SpacesObject Space
definition of objects. Also called Modeling space.
World Spacewhere the scene and viewing specification is made
Eye space (Normalized Viewing Space)where eye point (COP) is at the origin looking down the Z
axis.
3D Image SpaceA 3D Perspected space. Dimensions: -1:1 in x & y, 0:1 in Z. Where Image space hidden surface algorithms work.
Screen Space (2D) Coordinates 0:width, 0:height
![Page 20: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/20.jpg)
ProjectionsWe will look at several planar geometric 3D to 2D
projection:
-Parallel ProjectionsOrthographic Oblique
-Perspective
Projection of a 3D object is defined by straight projection rays (projectors) emanating from the center of projection (COP) passing through each point of the object and intersecting the projection plane.
![Page 21: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/21.jpg)
Perspective Projection
Perspective Projections
Distance from COP to projection plane is finite. The projectors are not parallel & we specify a center of projection.
Center of Projection is also called the Perspective Reference Point
COP = PRP
![Page 22: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/22.jpg)
• Perspective foreshortening: the size of the perspective projection of the object varies inversely with the distance of the object from the center of projection.
• Vanishing Point: The perspective projections of any set of parallel lines that are not parallel to the projection plane converge to a vanishing point.
![Page 23: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/23.jpg)
Example of Orthographic Projection
ProjectionPlane(front view)
Projectors for front view
ProjectionPlane(side view)
Projectors for top view
Projectorsfor side view
ProjectionPlane(top view)
![Page 24: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/24.jpg)
Example of Isometric Projection:
Projection-plane normal
Projectionplane
Projector
![Page 25: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/25.jpg)
Example Oblique Projection
Projectionplane
Projection-plane normal
Projectorz
y
x
![Page 26: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/26.jpg)
END OF BASICS
![Page 27: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/27.jpg)
THE CAMERA MODEL: perspective projection
p(x,y,z)
P (X,Y)
0
x,X
y,Y
z
I
f
COL
Camera lens
(x,y,z)- 3D world
(X,Y) - 2D Image plane
![Page 28: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/28.jpg)
F P(X,Y,Z)
Z
X or Y
IP
(COL)
PP
X or Y
xp or yp
P(X,Y,Z)
ZO
Perspective Geometry and Camera Models
![Page 29: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/29.jpg)
fz
yYf
zxX
f-zyfY,
zfxfX
fzy
fY
fzx
fX
−=
−=
=−
=
−−
=−−
=
1 ,
1
,• Image plane before the
camera lens
• Origin of coordinate systems at the image plane
• Image plane at origin of coordinate system
CASE 1
(COL) Z
X ,Y
P(-X,-Y)
p(x,y,z)
PP
f
O
x,y
By similarity of triangles
![Page 30: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/30.jpg)
fzyY
fzxX
zyfY,
zxfX
zy
fY
zx
fX
==
==
=−−
=−−
,
,
• Image plane before the camera lens
• Origin of coordinate systems at the camera lens
• Image plane at origin of coordinate system
CASE 2
(COL) Z
x ,y
P(-X,-Y)
p(x,y,z)
PP
f
O
X,Y
By similarity of triangles
![Page 31: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/31.jpg)
fzyY
fzxX
zyfY,
zxfX
zy
fY
zx
fX
==
==
==
,
,
• Image plane after the camera lens
• Origin of coordinate systems at the camera lens
• Focal length f
CASE 3
(COL)Z
x ,y
P(X,Y)
p(x,y,z)PP
fO
X, Y
By similarity of triangles
![Page 32: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/32.jpg)
fz
yYf
zxX
zfyfY,
zfxfX
zfy
fY
zfx
fX
+=
+=
+=
+=
+=
+=
1 ,
1
,• Image plane after the
camera lens
• Origin of coordinate system not at COP
• Image plane origin coincides with 3D world origin
CASE 4
(COL) Z
X ,Y
P(X,Y)
p(x,y,z)
PP
f O
x,y
By similarity of triangles
![Page 33: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/33.jpg)
(COP) Z
X ,Y
P(-X,-Y)
p(x,y,z)
PP
f
O
x,y
fz
yYf
zxX
−=
−=
1 ,
1
Consider the first case ….
• Note that the equations are non-linear
• We can develop a matrix formulation of the equations given below
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
kkzkykx
fkZYX
1100010000100001
'(Z is not important and is eliminated)
![Page 34: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/34.jpg)
Inverse perspective projection
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=−
1100010000100001
1
f
P
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
1000
1100010000100001
1
0
0
0
0
0
0
0
0
0
YX
k
kYkX
k
kYkX
fzyx
wh
p(x0,y0,z0)
P(X0,Y0)
Hence no 3D information can be retrieved with the inverse transformation
![Page 35: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/35.jpg)
)( ),( 00
000
0 zff
Yyzff
Xx −=−=
⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
+
+
+
=
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
+
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
1
1100010000100001
1
0
0
0
0
0
0
0
0
0
ZffZ
ZffY
ZffX
kfZk
kZkYkX
kkZkYkX
fzyx
wh
So we introduce the dummy variable i.e. the depth Z
Let the image point be represented as: TkkZkYkX ] [ 00
0
0
zffzZ−
= fzf
Zz
Zff 00 −
==+Zf
fZz+
=0
![Page 36: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/36.jpg)
fz
yYf
zxX
f-zyfY,
zfxfX
fzy
fY
fzx
fX
−=
−=
=−
=
−−
=−−
=
1 ,
1
,
CASE 1
(COL) Z
X ,Y
P(-X,-Y)
p(x,y,z)
PP
f
O
x,y
)( ),( 00
000
0 zff
Yyzff
Xx −=−=
Forward: 3D to 2D
Inverse: 2D to 3D
![Page 37: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/37.jpg)
fzyY
fzxX
zyfY,
zxfX
zy
fY
zx
fX
==
==
==
,
,
CASE 3
(COL)Z
x ,y
P(X,Y)
p(x,y,z)PP
fO
X, Y
Forward: 3D to 2D
Inverse: 2D to 3D
fYzy
fXzx 00
000
0. ,.
==
![Page 38: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/38.jpg)
Pinhole Camera schematic diagram
![Page 39: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/39.jpg)
Camera Image formulationAction of eye is simulated by an abstract camera model (pinhole camera model)3D real world is captured on the image plane. Image is projection of 3D object on a 2D plane.
XY
Z
worldX
imagexC
Π
f
),,( www ZYX=worldX
),(w
w
w
w
ZYf
ZXf=imagex
),(),,(: iiwww yxZYXF →
![Page 40: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/40.jpg)
Camera GeometryCamera can be considered as a projection matrix,
A pinhole camera has the projection matrix as
Principal point offset
Camera with rotation and translation
Xx 4*3P=
[ ]0|)1,,( IffdiagP =
( ) ( )TT /,/,, yx pZfYpZfXZYX ++→
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
000
10
0
00 y
x
pp
ff
K [ ]X0x |IK=
[ ]Xtx |RK=
p
camy
camx
),( yx pp=p
X
Y
Z
tR,
![Page 41: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/41.jpg)
Camera Geometry
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1yy
xx
pps
K αα
y
x
αα
Aspect ratio
xα
yα
Scale factor in x- coordinate direction
Scale factor in y- coordinate directions Camera skew
Camera internal parameters
Camera matrix, ]|[ tRKP = R RotationTranslation vectort
XY
Zp
imgximgy ),( yx pp=p
worldX
imagexC
Π
![Page 42: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/42.jpg)
Observations about Perspective projection
• 3D scene to image plane is a one to one transformation (unique correspondence)
• For every image point no unique world coordinate can be found
• So depth information cannot be retrieved using a single image ? What to do?
• Would two (2) images of the same object (from different viewing angles) help?
• Termed - Stereo Vision
![Page 43: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/43.jpg)
Stereo Vision
XY
XY
Image 1
Image 2B
Lens centerOptical axis
p(x,y,z)
(X2,Y2)
(X1,Y1)
World point
![Page 44: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/44.jpg)
Stereo Vision (2)• Stereo imaging involves obtaining two separate image
views of an object ( in this discussion the world point)• The distance between the centers of the two lenses is
called the baseline width.• The projection of the world point on the two image
planes is (X1, Y1) and (X2, Y2)• The assumption is that the cameras are identical • The coordinate system of both cameras are perfectly
aligned differing only in the x-coordinate location of the origin.
• The world coordinate system is also bought into the coincidence with one of the image X, Y planes (say image plane 1) . So y, z coordinates are same for both the camera coordinate systems.
![Page 45: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/45.jpg)
O1
B
(X2,Y2)
(X1,Y1)
f
f
W(x, y, z)
X
z1
Image 1
Image 2
Top view of the stereo imaging system with origin at center of first imaging plane.
O2
z2
![Page 46: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/46.jpg)
)( ),( 22
211
1 zff
Xxzff
Xx −=−=
First bringing the first camera into coincidence with the world coordinate system and then using the second camera coordinate system and directly applying the formula we get:
Because the separation between the two cameras is B
/* nowit Solve * /(?) , 2112 zzzBxx ==+=
)( ),()(
)( ),(
12
12
21
11
XXfBfzzf
fXXB
zff
XBxzff
Xx
−−=−
−=
−=+−=
![Page 47: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/47.jpg)
• The equation above gives the depth directly from the coordinate of the two points
• The quantity given below is called the disparity
)()( 12 zf
fBXXD−
=−=
• The most difficult task is to find out the two corresponding points in different images of the same scene – the correspondence problem.
]1[)( 12
DBf
XXfBfz −=−
−=
• Once the correspondence problem is solved –(non-analytical), we get D. Then obtain depth using:
![Page 48: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/48.jpg)
.(?) ; , 212112 zzzyyyBxx ====−=
DfB
XXfBz
fzXXB
fzXBxx
fzXx
.)(
,)(
,
21
21
212
11
=−
=−
=
=−==
fYzy,
fXzx
zy
fY
zx
fX
==
==
,Alternate Model – Case III
![Page 49: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/49.jpg)
O1
B
(X2,Y2)
(X1,Y1)
fW(x, y, z)
X
z1
IP 1
IP 2
Top view of the stereo imaging system with origin at center of first camera lens.
O2
z2
![Page 50: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/50.jpg)
]1[)( 12
DBf
XXfBfz −=−
−=
DfB
XXfBz .
)( 21
=−
=
Compare the two solutions
What do you think of D ?
)()( 12 zf
fBXXD−
=−=
zfBXXD =−= )( 21
![Page 51: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/51.jpg)
The Correspondence Problem
X2
Y2Y1
X1
21 YY =zfBXXD =−= )( 21
12 ;0 XXthenDIf <>
Image Plane - I Image Plane - II
DfBz .=
EPIPOLAR Line
(X1, Y1) (X2, Y2)
![Page 52: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/52.jpg)
Error in Depth Estimation
DfBz .= 2
.D
fB−=Dzδ
δ )(
fBz
Dz
DfB
Dz
..)( 2
2 −=−=−=δδ
Expressing in terms of depth (z), we have:
What is the maximum value of depth (z), you can measure using a stereo setup ?
fBz .max =
![Page 53: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/53.jpg)
Even if correspondence is solved correctly, the computation of D may have an error, with an upper bound of 0.5; i.e. (δD)max = 0.5.
That may cause an error of:fB
zz.2
)(2
−=δ
Larger baseline width and Focal length (of the camera) reduces the error and increases the maximum value of depth that may be estimated.
What about the minimum value of depth (object closest to the cameras) ?
maxmin /. DfBz =What is Dmax ?
maxmax XD =Xmax depends on f and image resolution (in other words, angle of field-of-view or FOV).
![Page 54: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/54.jpg)
![Page 55: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/55.jpg)
![Page 56: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/56.jpg)
General Stereo Views
![Page 57: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/57.jpg)
Perfect Stereo Views
![Page 58: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/58.jpg)
Perfect Stereo Views
![Page 59: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/59.jpg)
Perfect Stereo Views
![Page 60: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/60.jpg)
![Page 61: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/61.jpg)
We can also have arbitrary pair of views from two cameras.
• The baseline may not lie on any of the principle axis
• The viewing axes of the cameras may not be parallel
• Unequal focal lengths of the cameras
• The coordinate systems of the image planes may not be aligned
In general we may have multiple views ( 2 or more) of a scene. Typically used for 3D surveillance tasks.
Take home exercises/problems:
What about Epipolar line in cases above ?
How do you derive the equation of an epipolar line ?
![Page 62: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/62.jpg)
In case of a set of arbitrary views used for 3-D reconstruction (object structure, surface geometry, modeling etc.), methods used involve:
- KLT (Kanade-Lucas-Tomasi)- tracker
- Bundle adjustment
- 8-point DLT algorithm
- Zhang’s homography
- Tri-focal tensors
- Cheriality and DIAC
- Auto-calibration
- Metric reconstruction
- RANSAC
![Page 63: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/63.jpg)
Tri-focal tensors
![Page 64: Computer Vision – Transformations, Imaging Geometry and Stereo …vplab/courses/CV_DIP/PDF/GEOM+PROJ+STERE… · Computer Vision – Transformations, Imaging Geometry and Stereo](https://reader036.fdocuments.us/reader036/viewer/2022070613/5b87ff707f8b9a301e8c9225/html5/thumbnails/64.jpg)
End of Lectures on -
Transformations, Imaging Geometry
andStereo Vision