Post on 16-Jan-2016
3D-2D registration
Kazunori UmedaChuo Univ., Japan
umeda@mech.chuo-u.ac.jp
http://www.mech.chuo-u.ac.jp/umedalab/
CRV2010 TutorialMay 30, 2010
Registrationof range image and color image
Necessary for texture mapping
Range image(3D model)
Color image
When 3D-2D registration is given,
Texture mapping
Parameters to obtain for 3D-2D registration
Color camera
Image plane
Object(range image)
Range imagesensor
Sensorcoordinate system
Projection of range intensity image
Intensity image
Intrinsic parameters
Extrinsic parameters (Distortion parameters)
X
Y
Z
Extrinsic parametersObject’ rotation R and translation t or camera’s orientation Rc and position tc
Color camera
Range imagesensor
Sensorcoordinate system
X
Y
Z
R, t (Rc, tc)
z
y
x
t
t
t
rrr
rrr
rrr
R t,
333231
232221
131211
Parameters to obtain for 3D-2D registration
tt Tc
Tc RRR ,
Intrinsic parameters
(X,Y,Z) (u,v)3D space Image plane
cameracoordinate system
0
0
vZ
Yv
uZ
sYXu
v
u
00 ,,,, vusvu
u, v: focal length/pixel sizes: skew, u0, v0: principal point coordinates
Color camera
Image plane
u
v
),,( ZYX
),( vu
uxX
Parameters to obtain for 3D-2D registration
Homogenous coordinates
100
0
111
0
0
v
us
A
Z
Y
X
PZ
Y
X
RAv
u
s
v
u
w
w
w
w
w
w
t
Parameters to obtain for 3D-2D registration
P: 34 matrix
11 unknown parameters (6 extrinsic + 5 intrinsic)2 constraints
When correspondences between range image and color image are given,
…It is hard to obtain correspondences even manually.
Parameters can be calculated.Equivalent to camera calibration problem.For extrinsic parameter estimation,
Equivalent to PnP (Perspective n-Point) problem
6n
3n
3D 2D
Range image
Range intensity image(reflectance image)
By using range intensity image, obtaining correspondences becomes easier!
e.g., corners, edges,SIFT [Böhm 2007]
Our approach: gradient-based method(not explicitly using correspondences)
Two 2Dimages are matched
Gradient-based method
Update camera parameters
Produce a 2D imagefrom a range image
Initial camera parameters
End
Yes
No
uv
Projection of range intensity image
Intensity image
),( vu
),( vu
tvu IvIuI
t
II
v
II
u
II tvu
,,
Optical flow constraint
It: difference between intensity image and projected range intensity image
),,(),,( tvuIttvvuuI Tailor expansion
0
0
vZ
Yv
uZ
sYXu
v
u
(1) Constraints for extrinsic parameters
When intrinsic parameters are constant,
ZZ
YY
Zv
ZZ
sYXY
Z
sX
Zu
vv
uu
2
2
Substituting for the optical flow constraint
tvu IvIuI
tv
vu
uv
vuu
u IZZ
YI
Z
sYXIY
ZI
Z
sIX
ZI
22
XωvX 0 Camera motion: v0,
TZYXX
TzyxT
zyx vvv ωv0 ,000Digital camera
uv
),,( ZYX
),( vu
22
000
,,
,)()()(
Z
YI
Z
sYXIc
ZI
Z
sIb
ZIa
IbXaYaZcXcYbZcvbvav
vv
uu
vvu
uu
tzyxzyx
Linear equation for 6 motion parameters v0,
v0, can be solved with 6 or more points by linear least square method.
Motion parameters are supposed to be smallIteration is necessary
v0, R(33 rotation matrix) and t (3D translation vector)
[Yamamoto 1985][Horn IJCV1988]
cf.
(2) Constraints for intrinsic parameters
0
0
vZ
Yv
uZ
sYXu
v
u
When intrinsic parameters are also variables,
02
02
vZ
YZ
Z
YY
Zv
usZ
Y
Z
XZ
Z
sYXY
Z
sX
Zu
vvv
uuu
Substituting for the optical flow constraint
tvu IvIuI
tvuuvvuu
zyxzyx
IvIuIsZ
YI
Z
YI
Z
XI
bXaYaZcXcYbZcvbvav
00
000 )()()(
a,b,c: same as previous equation
Linear equation for 6 motion parameters v0, and5 intrinsic parameters
v0, and intrinsic parameters can be solved with 11 or more points by linear least square method.
(3) Constraints for distortion
0
0
vZ
Yv
uZ
sYXu
v
u
02
22
1
02
22
1
1
1
vZ
YXk
Z
Yv
uZ
YXk
Z
sYXu
v
u
Distortion model (the simplest)
)'1(
)'1(2
1
21
rkyy
rkxx
d
d22 yxr
uxxX d
Distortion
4
22
124
22
12
3
22
13
22
1
313
22
1
)(3))((3
,)3(2)3(
,22)3(
Z
YXYk
Z
YI
Z
YXsYXk
Z
sYXIc
Z
YXk
ZI
Z
XYYXsk
Z
sIb
Z
XYkI
Z
sXYYXk
ZIa
vvv
uuu
vvv
uu
vv
uuu
tv
vu
uvu
uvvuu
zyxzyx
IkZ
YXYI
Z
YXsYXIvIuI
sZ
YXYk
Z
YI
Z
YXYk
Z
YI
Z
YXXk
Z
XI
bXaYaZcXcYbZcvbvav
13
22
3
22
00
3
221
3
221
3
221
000
)())((
)()()(
)()()(
Linear equation for 6 motion parameters v0, , 5 intrinsic parameters and a distortion parameter
The parameters can be solved with 12 or more points by linear least square method.
Implementation
Differential imagesSo as to absorb the differences between a range intensity image and an intensity image2 images: horizontal, vertical. Prewitt operator.
-1-1-1
111
000
-101
-101
-101
Coarse to fineControl of resolution and of GaussianExtrinsic onlyv0+Intrinsicall
[Irani ICCV1998]
Experimental results
Range image sensor: ShapeGrabber PLM300 ( Slit laser, triangulation , wavelength 670nm)
R-channel , RAW format256019201280960 at registration
Digital camera: Nikon COOLPIX 5000(5M, 25601920 pixels,2/3” CCD, pixel dimension 3.4m?, f=7.1-21.4mm)
Measurement of a range image
312730 points
Summary
3 D-2D registration (for texture mapping, etc.)
•Projective geometry• Obtaining camera’s extrinsic and intrinsic parameters
•Range intensity (reflectance) image is useful•With correspondences
• Equivalent to {camera calibration / PnP} problems•Using optical flow constraint
• Explicit correspondences are not necessary• Linear equation for motion parameters