Kosuke Takahashi, Akihiro Miyata, Shohei Nobuhara, Takashi … · 2018. 1. 23. · Key Idea:...
1
is expressed as the first reflec/on of on . A linear constraint on is hold as follow, A Linear Extrinsic Calibration of Kaleidoscopic Imaging System from Single 3D Point Kosuke Takahashi , Akihiro Miyata, Shohei Nobuhara, Takashi Matsuyama (Kyoto Univ.) Background • Bioinformatics • The marine products industry 3D reconstruction and motion analysis of micro-scale objects in water. Final goal : 3D Reconstruc/on of micro-scale objects Our goal is 3D reconstruction of Ø Object ：micro-scale object (An embryo，a water flea, etc) Ø Scale ：mm, nm, μm Ø Device ：microscope + camera. In this work, we utilize a virtual multi-view system with planar mirrors ( KIS : Kaleidoscopic Imaging System ) Microscope Water Mirror Vritual microscope Mirror Outline of this work This paper is aimed at proposing a new method of extrinsic calibration of KIS, i.e. estimation of mirror normals and distances . Zhang utilizes a reference object of known geometry such as a chessboard. It is difficult to prepare such reference object of known geometry in micro-scale. This paper proposes a new linear extrinsic calibration method from a single 3D point of unknown geometry. KIS : Kaleidoscopic Imaging System n i d i It is difficult to capture mul/-view images with mul/ple microscopes due to physical constraints Purpose Proposed method Es/ma/on of mirror normals Es/ma/on of mirror distances Bundle Adjustment INPUT Projec/ons of single 3D point and its reflec/ons OUTPUT Mirror parameters 2 4 y 0 - y 1 x 1 - x 0 x 0 y 1 - x 1 y 0 y 2 - y 12 x 12 - x 2 x 2 y 12 - x 12 y 2 y 3 - y 13 x 13 - x 3 x 3 y 13 - x 13 y 3 3 5 n 1 = 0 3⇥1 p p 0 ⇡ n d q q 0 Camera A , A -1 q =(x, y, 1) > has 2 DOF q 0 q 1 q 2 q 12 q 3 q 13 p 0 p 1 ⇡ 1 ⇡ 2 ⇡ 3 Mirror Mirror Mirror p 12 p 2 p 13 p 3 = S 1 p 2 Reflection on p 12 p 2 , provide similar constraints as well. p 3 p 13 can be computed from single 3D point and its reflections. n Second reflec/on Non-linear Linear p q ⇡ i ⇡ ij d i d ij q i q ij n i n ij p i p ij [First reflection] (A -1 q i ) ⇥ p i = x i ⇥ p i = 0 3⇥1 (A -1 q ij ) ⇥ p ij = x ij ⇥ p ij = 0 3⇥1 [Second reflections] K [p 0 d 1 d 2 d 3 ] > = 0 30⇥1 Linear Minimizing nonlinearly over . E (n 1 , n 2 , n 3 ,d 1 ,d 2 ,d 3 )= [q 0 - ˆ q 0 , e 1 , e 2 , e 3 , e 0 1,2 , e 0 2,3 , e 0 3,2 , e 0 3,1 , e 0 1,3 ] p 12 = S 12 p 1 = S 12 S 1 p 0 = S 1 S 2 p 0 can be estimated by using more than or equal two reference points. ⇡ 1 ||E (·)|| 2 n 1 , n 2 , n 3 ,d 1 ,d 2 ,d 3 (y 2 - y 12 x 12 - x 2 x 2 y 12 - x 12 y 2 )n 1 =0 Epipolar geometry with a planar mirror n n (n ⇥ p) > p 0 =0 q > A > [n] > ⇥ A -1 q 0 =0 (y - y 0 x 0 - x xy 0 - x 0 y )n =0 , is intrinsic parameters Key Idea: U/lizing 2D projec/ons of mul/ple reflec/ons to form a linear system on mirror parameters Conven/onal method Problem Contribu/on Aqua Vision Many applica/ons Colinearity constraint p 0 p 1 p 12 ⇡ 1 Reflection on ⇡ 12 First reflec/on Linear ⇡ 1 Implicitly requests to be consistent with the other mirrors. Experiments: SimulaCon Evaluation of our proposed method under the observation noise. [Baseline] Zhang’s method with a chessboard. 0 0.5 1 1.5 2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 #10 -3 0 0.5 1 1.5 2 0 1 2 3 4 5 6 #10 -3 0 0.5 1 1.5 2 0 0.5 1 1.5 2 2.5 Error of normals (rad) Error of distances (mm) Reprojection error (pixel) Horizontal axis：The standard deviation of Gaussian noise (pixel). Vertical axis：Estimation error of each parameter. Proposed method (5 points) + BA Baseline (5 points) + BA Proposed method (5 points) Proposed method (1 point) + BA Proposed method (1 point) Camera Setup ApplicaCon: 3D reconstrucCon Results Mirror normals and distances Proposed method (5 points) + BA Baseline (5 points) + BA We can conclude that the proposed method (1) outperforms baseline method even without final BA. (2) can achieve comparable estimation linearly even with a single point. While the estimated mirror parameters look close to each other, the reprojection error of the proposed method is smaller than that of the baseline method. Experiments: Real data Result Calibration Method ReprojecCon Error (pixel) Proposed + BA 3.37 Baseline 4.75 Camera Projector Setup Right figure shows a 3D rendering of the estimated 3D shape using the mirror parameters estimated by the proposed method. These results show the proposed method provides a sufficiently accurate calibration for 3D shape reconstruction. Code Available http://vision.kuee.kyoto-u.ac.jp/~nob/proj/kaleidoscope/ Evaluation of our proposed method with real data. [Baseline] Zhang’s method with a chessboard. Camera Mirror Virtual camera n 1 n 2 n 3 d 1 d 2 d 3 A reference point (unknown) Reflections of a reference point n i d i Summary This paper proposes a new linear extrinsic calibration method of kaleidoscopic imaging system from a single 3D point of unknown geometry. INPUT Projec/ons of single 3D point and its reflec/ons OUTPUT Mirror normals and distances . d i n i n i d i Proposed Method (Image size: 6000x4000 pixel)
Transcript of Kosuke Takahashi, Akihiro Miyata, Shohei Nobuhara, Takashi … · 2018. 1. 23. · Key Idea:...
isexpressedasthefirstreflec/onofon.Alinearconstraintonisholdasfollow,
A Linear Extrinsic Calibration of Kaleidoscopic Imaging System from Single 3D Point Kosuke Takahashi, Akihiro Miyata, Shohei Nobuhara, Takashi Matsuyama (Kyoto Univ.)
Background
• Bioinformatics • The marine products industry
3D reconstruction and motion analysis of micro-scale objects in water.
Finalgoal:3DReconstruc/onofmicro-scaleobjectsOur goal is 3D reconstruction of
Ø Object ：micro-scale object (An embryo，a water flea, etc) Ø Scale ：mm, nm, µm Ø Device ：microscope + camera.
In this work, we utilize a virtual multi-view system with planar mirrors ( KIS : Kaleidoscopic Imaging System )
Microscope
Water
Mirror
Vritual microscope
Mirror
Outlineofthiswork
This paper is aimed at proposing a new method of extrinsic calibration of KIS, i.e. estimation of mirror normals and distances .
Zhang utilizes a reference object of known geometry such as a chessboard.
It is difficult to prepare such reference object of known geometry in micro-scale.
This paper proposes a new linear extrinsic calibration method from a single 3D point of unknown geometry.
KIS : Kaleidoscopic Imaging System
ni di
Itisdifficulttocapturemul/-viewimageswithmul/plemicroscopesduetophysicalconstraints
Purpose
Proposedmethod
Es/ma/onofmirrornormals
Es/ma/onofmirrordistances
BundleAdjustment
INPUTProjec/onsofsingle3Dpoint
anditsreflec/ons
OUTPUTMirrorparameters
2
4y0 � y1 x1 � x0 x0y1 � x1y0
y2 � y12 x12 � x2 x2y12 � x12y2
y3 � y13 x13 � x3 x3y13 � x13y3
3
5n1 = 03⇥1
p
p0
⇡
nd
q
q0
Camera
A
,A
�1q = (x, y, 1)>
has 2 DOF
q0
q1
q2
q12q3
q13
p0
p1
⇡1 ⇡2
⇡3
Mirror Mirror
Mirror
p12
p2
p13
p3
= S1p2
Reflection on
p12 p2
, provide similar constraints as well.p3 p13
can be computed from single 3D point and its reflections.n
Secondreflec/onNon-linear
Linear
p
q
⇡i
⇡ij
di
dij
qi
qijni
nijpi
pij
[First reflection](A�1qi)⇥ pi = xi ⇥ pi = 03⇥1
(A�1qij)⇥ pij = xij ⇥ pij = 03⇥1
[Second reflections]
K[p0 d1 d2 d3]> = 030⇥1 Linear
Minimizing nonlinearly over . E(n1,n2,n3, d1, d2, d3) = [q0 � q̂0, e1, e2, e3, e
01,2, e
02,3, e
03,2, e
03,1, e
01,3]
p12 = S12p1
= S12S1p0 = S1S2p0
can be estimated by using more than or equal two reference points.
⇡1
||E(·)||2 n1,n2,n3, d1, d2, d3
(y2 � y12 x12 � x2 x2y12 � x12y2)n1 = 0
Epipolargeometrywithaplanarmirror
n
n
(n⇥ p)>p0 = 0
q>A>[n]>⇥A�1q0 = 0
(y � y
0x
0 � x xy
0 � x
0y)n = 0,
is intrinsic parameters
KeyIdea:U/lizing2Dprojec/onsofmul/plereflec/onstoformalinearsystemonmirrorparameters
Conven/onalmethod
Problem
Contribu/on
AquaVision Manyapplica/ons
Colinearityconstraint
p0 p1 p12⇡1 Reflection on ⇡12
Firstreflec/onLinear
⇡1
Implicitlyrequeststobeconsistentwiththeothermirrors.
Experiments:SimulaCon
Evaluation of our proposed method under the observation noise. [Baseline] Zhang’s method with a chessboard.
0 0.5 1 1.5 20
0.2
0.4
0.6
0.8
1
1.2
1.4#10-3
0 0.5 1 1.5 20
1
2
3
4
5
6 #10-3
0 0.5 1 1.5 20
0.5
1
1.5
2
2.5
Error of normals (rad) Error of distances (mm) Reprojection error (pixel)
Horizontal axis：The standard deviation of Gaussian noise (pixel). Vertical axis：Estimation error of each parameter.
Proposed method (5 points) + BA
Baseline (5 points) + BA
Proposed method (5 points)
Proposed method (1 point) + BA
Proposed method (1 point)
CameraSetup
ApplicaCon:3DreconstrucCon
Results
Mirrornormalsanddistances
Proposed method (5 points) + BA
Baseline (5 points) + BA
We can conclude that the proposed method (1) outperforms baseline method even without final BA. (2) can achieve comparable estimation linearly even with a single point.
While the estimated mirror parameters look close to each other, the reprojection error of the proposed method is smaller than that of the baseline method.
Experiments:Realdata
Result
Calibration
Method ReprojecConError(pixel)Proposed+BA 3.37Baseline 4.75
CameraProjector
Setup
Right figure shows a 3D rendering of the estimated 3D shape using the mirror parameters estimated by the proposed method. These results show the proposed method provides a sufficiently accurate calibration for 3D shape reconstruction.
CodeAvailable http://vision.kuee.kyoto-u.ac.jp/~nob/proj/kaleidoscope/
Evaluation of our proposed method with real data. [Baseline] Zhang’s method with a chessboard.
CameraMirror
Virtual camera n1n2
n3
d1 d2
d3
A reference point (unknown)
Reflections of a reference point ni di
Summary This paper proposes a new linear extrinsic calibration method of kaleidoscopic imaging system from a single 3D point of unknown geometry. INPUT
Projec/onsofsingle3Dpointanditsreflec/onsOUTPUT
Mirrornormalsanddistances.
di
ni
ni di
ProposedMethod
(Image size: 6000x4000 pixel)
