What does calibration give? An image line l defines a plane through the camera center with normal...
-
Upload
britton-byrd -
Category
Documents
-
view
217 -
download
0
Transcript of What does calibration give? An image line l defines a plane through the camera center with normal...
What does calibration give?
xKd 1
0d0]|K[Ix
21-T-T
211-T-T
1
2-1-TT
1
2T
21T
1
2T
1
)xK(Kx)xK(Kx
)xK(Kx
dddd
ddcos
An image line l defines a plane through the camera center
with normal n=KTl measured in the camera’s Euclidean
frame. In fact the backprojection of l is PTl n=KTl
The image of the absolute conic
KRd0
dO]|KR[IPXx
mapping between ∞ to an image is given by the planar
homogaphy x=Hd, with H=KR
absolute conic (IAC), represented by I3 within ∞ (w=0)
1-T-1T KKKKω 1TCHHC
(i) IAC depends only on intrinsics(ii) angle between two rays(iii) DIAC=*=KKT
(iv) K (Cholesky factorization)(v) image of circular points belong
to (image of absolute conic)
2T
21T
1
2T
1
ωxxωxx
ωxxcos
its image (IAC)
A simple calibration device
(i) compute Hi for each square
(corners (0,0),(1,0),(0,1),(1,1))
(ii) compute the imaged circular points Hi [1,±i,0]T
(iii) fit a conic to 6 imaged circular points
(iv) compute K from K-T K-1 through Cholesky factorization (= Zhang’s calibration method)
Orthogonality relation
2T
21T
1
2T
1
ωvvωvv
ωvvcos
0ωvv 2T
1
0lωl 2*T
1
Calibration from vanishing points and lines
Calibration from vanishing points and lines
Two-view geometry
Epipolar geometry
F-matrix comp.
3D reconstruction
Structure comp.
(i) Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the
corresponding point x’ in the second image?
(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?
(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?
Three questions:
The epipolar geometry
C,C’,x,x’ and X are coplanar
The epipolar geometry
What if only C,C’,x are known?
The epipolar geometry
All points on project on l and l’
The epipolar geometry
Family of planes and lines l and l’ Intersection in e and e’
The epipolar geometry
epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction
an epipolar plane = plane containing baseline (1-D family)
an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)
Example: converging cameras
Example: motion parallel with image plane
The fundamental matrix F
algebraic representation of epipolar geometry
l'x
we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F
The fundamental matrix F
algebraic derivation
λC0
xMλX
-1
1MM'e'F
xMM'Cm'M'0
xMm''MC'm'Ml' 1
1
e’ is the image of C taken by the second camera
xP
λX0Fxx'0 l' x' l' x' T
The fundamental matrix F
correspondence condition
0Fxx'T
The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images 0l'x'T
The fundamental matrix F
F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’
(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)
(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x
e’TF=0, similarly Fe=0;( vi e’ is left null space of F(v) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(vi) F is a correlation, projective mapping from a point x to
a line l’=Fx (not a proper correlation, i.e. not invertible)
Projective transformation and invariance
-1-T FHH'F̂ x'H''x̂ Hx,x̂
Derivation based purely on projective concepts
X̂P̂XHPHPXx -1
F invariant to transformations of projective 3-space
X̂'P̂XHHP'XP'x' -1
FP'P,
P'P,F
unique
not unique
canonical form
m]|[MP'0]|[IP
MmF
Projective ambiguity of cameras given Fprevious slide: at least projective ambiguitythis slide: not more!
Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’
~ ~
~ ~
]a~|A~
['P~ 0]|[IP
~ a]|[AP' 0]|[IP
A~
a~AaF
T1 avAA~
kaa~ klemma:
kaa~Fa~0AaaaF2rank
TavA-A~
k0A-A~
kaA~
a~Aa
kkIkH T1
1
v0
'P~
]a|av-A[v
0a]|[AHP' T1T1
1
kk
kkIk
(22-15=7, ok)
Canonical cameras given F
F matrix corresponds to P,P’ iff P’TFP is skew-symmetric
X0,FPXP'X TT
F matrix, S skew-symmetric matrix
]e'|[SFP' 0]|[IP
000FSF
0Fe'0FSF0]|F[I]e'|[SF
TT
T
TTT
(fund.matrix=F)
Possible choice:
]e'|F][[e'P' 0]|[IP
Canonical representation:
]λe'|ve'F][[e'P' 0]|[IP T
Triangulation of points in 3D
Ransac: - 8 point correspondence samples
Computation of F1i
-Compute viewing rays from cameras-Intersect viewing rays associated to corresponding points
Computation of canonical camerasfrom F1i
PROJECTIVE RECONSTRUCTION
The essential matrix
~fundamental matrix for calibrated cameras (remove K)
t]R[RRtE T
0x̂E'x̂ T
FKK'E T
x'K'x̂ x;Kx̂ -1-1
5 d.o.f. (3 for R; 2 for t up to scale)
E is essential matrix if and only iftwo singularvalues are equal (and third=0)
T0)VUdiag(1,1,E SVD
100
001
010
WGiven
Motion from E
TTVUWR'
TUWV'R'
and
3Ut'
3U't'
Four solutions
Four possible reconstructions from E
(only one solution where points is in front of both cameras)
Self-calibration
Motivation
• Avoid explicit calibration procedure– Complex procedure– Need for calibration object – Need to maintain calibration
Motivation
• Allow flexible acquisition– No prior calibration necessary– Possibility to vary intrinsics– Use archive footage
Constraints ?
• Scene constraints– Parallellism, vanishing points, horizon, ...– Distances, positions, angles, ...
Unknown scene no constraints
• Camera extrinsics constraints– Pose, orientation, ...
Unknown camera motion no constraints • Camera intrinsics constraints
– Focal length, principal point, aspect ratio & skew
Perspective camera model too general some constraints
Constraints on intrinsic parameters
Constant e.g. fixed camera:
Knowne.g. rectangular pixels:
square pixels:
principal point known:
21 KK
0s
1yy
xx
uf
usf
K
0, sff yx
2,
2,
hwuu yx
Self-calibration
Upgrade from projective structure to metric structure using constraints on intrinsic camera parameters– Constant intrinsics
– Some known intrinsics, others varying
– Constraints on intrincs and restricted motion(e.g. pure translation, pure rotation, planar motion) (Moons et al.´94, Hartley ´94, Armstrong ECCV´96)
(Faugeras et al. ECCV´92, Hartley´93,
Triggs´97, Pollefeys et al. PAMI´98, ...)
(Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...)
A counting argument
• To go from projective (15DOF) to metric (7DOF) at least 8 constraints are needed
• Minimal sequence length should satisfy
• Independent of algorithm• Assumes general motion (i.e. not critical)
8#1# fixednknownn
known internal parameters
-1-TKKω
0ωω 2112 0s
rectangular pixels
yx ff
square pixels
2211 ωω
Same camera for all images
same intrinsics same image of the absolute conic
e.g. moving cameras
given sufficient images there is in general only oneconic that projects to the same image in all images,i.e. the absolute conic
This approach is called self-calibration, see later
ωHHω' Ttransfer of IAC:
(a) Compute F from xi↔x‘i(b) Compute P,P‘ from F(c) Triangulate Xi from xi↔x‘i
Obtain projective reconstruction (P,P‘,{Xi})
0]|[IP1 ]a|[AP iii
e.g. ]e|veFe[P iT
ixii
t]|[A1v
tA0]|[I0]H|[I0]|[KP T1M1
The metric reconstruction will be
t=0 1KA
Let the unknown plane at the infinity in the starting projective reconstruction be
1
pπ*
From1
pπH
1ππ *T*
0M
1
vK-
110
vK-K-
1H
1
pπ
-T1
-T1
-T1T-*
00
thus
v-Kp -T1 and p-Kv T
1Therefore,
1Kp
0KH
1T
1
iiiii1T
iiiiM tKRKa)Kpa-(AHaAPi
Now
8 unknowns:
Tiiii
Tii )R)(KR(KKK
TTii
T11
Tii
Tii )pa-(AK)Kpa-(AKK
Therefore, since
Self-calibration equation
1K and p
Equations come from constraints on intrinsics:-known parameters (e.g. s=0 or square pixels)-fixed parameters (e.g. s, or aspect ratio)
8#1# fixednknownn
Solution: nonlinear algorithms (Cipolla)
needed views: n, with
Uncalibrated visual odometry Uncalibrated visual odometry for ground plane motionfor ground plane motion
(joint work with Simone Gasparini)
Problem formulation
• Given:– an uncalibrateduncalibrated camera mounted on a robot– the camera is fixedfixed and aims at the floor– the robot moving on a planarplanar floor (ground
plane)
• Determine:– the estimate of robot motion from observed
features on the floor
Technique involvedEstimate the ground plane transformation (homography) between images taken before and after robot displacement
Motivations• Dead reckoning techniques are not reliable and diverge after few steps
[Borenstein96]
• Visual odometry techniques exploits cameras to recover motion
• We use a single uncalibrated camera– 3D reconstruction with uncalibrated camera usually require auto-calibration– Non-planar motion is required [Triggs98]
– Planar motion with different camera attitudes [Knight03]
– Special devices is required (e.g. PTZ cameras)
• Stereo cameras approaches [Nister04, Takaoka04, Agrawal06, Cheng06]
• Catadioptric cameras approaches [Bunschoten03, Corke04]
– Assume Single View Point (difficult setup)
• Our method similar in spirit to [Wang05] and [Benhimane06]
– We do not assume camera calibration
Problem formulation
• Fixed uncalibrated camera mounted on a robot
• The pose of the camera w.r.t. the robot is unknown
• The projective transformation between ground plane and image plane is a homography T (3x3)
• T does not change with robot motion
• T is uknownT is uknown
Image plane
Ground Plane
X
x x=TX
Problem formulation
• Robot and camera undergo a planar motion– rotation of unknown angle about unknown
vertical axis– 2D rotation matrix R (3x3 with homogeneous
coordinates)
2x2ˆ ˆ orthogonal matrix
0 translation vector (3x1 )
R R
R tt
Problem formulation
• Given 2 images before and after robot displacement determine:– Rotation centre C– Rotation angle – Determine unknown transformation T between
ground plane and image plane
Ground reference frame
• We define a ground reference frame– O is the origin of the projected reference frame
• E.g. O is the backprojection of image point O’(0,0)
– The vector connecting the origin to a point A is the unit vector along the x-axis
• E.g. A is the backprojection of image point A’(100,0)
Image plane
Ground Plane
O
A
O'
A’
Estimation of robot displacement
• Transformation relating the two images is still a homography
• Eigenvectors of H:
1 ( unknown)H = TRT T
' associated to the eigval, C rotation center
' associated to the eigval, I and J circular points
'
C TC
eigvec I TI
J TJ
H =
• Since eig(H)=eig(R)– Angle can be calculated through the ratio of
imaginary and real part of the complex eigenvalue of H
– E.g. the eigenvalue associated to I’ is e±i
Estimation of robot displacement: degenerate motion
Pure translation: a frequent motion on planar floor
• Under pure translation, the whole line at the infinity is invariant
ground plane can be only rectified modulo an affine transformation: e.g., orientation of translaton wrt ground reference
can not be determined
• In practice: – Use a motion including rotations to estimate
ground plane to image plane transformation T– Use knowledge of T while translating
Estimation of the transformation T
• Estimating T allows to determine the shape
of observed features
• Four pairs of corresponding points are
needed
– The 2 circular points I and J
– The 2 points A and O used for ground reference
where 1, ,0
where 1, ,0
where 0,0,1 and ' 0,0,1
where 1,0,1 and ' 100,0,1
T
T
T T
T T
i
i
I' = TI I
J' = TJ J
O' = TO O O
A' = TA A A
Experimental set up
• Fixed perspective camera placed on turntable ( ground truth)
• Optical distortion is negligible
• Camera pointing towards the ground floor
• Camera viewpoint in a generic position wrt rotation axis
• Basic algorithm:– Feature (corner) extraction from floor texture
[Harris88]– Feature tracking and matching in order to
determine correspondences– Corresponding features used to fit the
homography H– Eigendecomposition of matrix H
• Rotation angle• Image of circular points• Image of rotation centre• Ground plane to image plane transformation
Experiments on sequences of images
Sequence of large displacements
• Image displacements of the order of 10°
• Features extracted and matched [Torr93]
• Use best matching pairs to fit a homography using RANSAC [Fischler81]
• Good overall accuracy (error < 1°)
• Larger displacements affect matching
Experiments on sequences of images
Sequence of small displacements
• Image displacements of about 5°
• Small rotations may lead to numerical instability
• Track features over three images– H13 fitted with correspondences of 1 and 3
• Good overall accuracy (error < 1°)
HH1313
1 2 3
Experiments on a mobile platform
Feature extraction and matching between images
Rotation and relevant centre of rotation determination