Multi video camera calibration and synchronization
description
Transcript of Multi video camera calibration and synchronization
![Page 1: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/1.jpg)
Multi video camera calibration and synchronization
![Page 2: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/2.jpg)
Motivation
• Multi camera applications become common.Example: Stereo, Surveillance….
• Using multi camera we can over come problems like hidden objects.
• In general more cameras equal more information on the scene.
![Page 3: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/3.jpg)
How does it look
• Multi camera setup
![Page 4: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/4.jpg)
The scene
• The filmed scene 1/3
![Page 5: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/5.jpg)
The scene
• The filmed scene 2/3
![Page 6: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/6.jpg)
The scene
• The filmed scene 3/3
![Page 7: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/7.jpg)
Perspective projection
Perspective projection:
Projection plane
P(x, y, z)
P'(xp, yp, zp)
d
Zv
Xv
Yv
P(x, y, z)
P(x, y, z)
Xp
Xv
Yv
Zv
Zv
yp
Im * objectage
object
Xx d
Z
Im * objectage
object
Yy d
Z
![Page 8: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/8.jpg)
The projection matrix
• Object point • Image point• Using the model:
• And the projection matrix (so far) is:
*Homogenous coordinates
( , , )TP X Y Z( , )Tp x y
( , , ) ( , ) ( * , * )object objectT T T
object object
X YX Y Z x y d d
Z Z
0 0 00 0 0
1 0 0 1 01
Xx d
Yy d
Z
![Page 9: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/9.jpg)
Internal matrix
• The internal matrix represent the inner camera settings
• Focal length (d)• Principle point location usually
(0,0)• Scaling factor
( , )x yo o
( , )x ys s
![Page 10: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/10.jpg)
External matrix
• Includes all the orientation properties of the camera
• Rotation• Translation
( )camera worldp R p T
![Page 11: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/11.jpg)
Projection matrix sum up
• Internal parameters
• External parameters
• The result p=MP
int
0 00 00 0 1 0
x x
ernal y y
a oM a o
int *ranal externalM M M
11 12 13 1
21 22 23 2
31 32 33 3
0 0 0 1
extenal
R R R RTR R R RT
MR R R RT
![Page 12: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/12.jpg)
Calibration
• Camera calibration is used to coordinate between cameras.
• Given a 3D point in the real word finding the projected point in the camera.
• The goal is to fined the projection matrix M.• Using known 3D points and there
corresponding image points p=MP can be solved.
![Page 13: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/13.jpg)
When a full calibration is not necessary Homography
• Mapping between a point on a ground plane as seen from one camera, to the same point on the ground plane as seen from a second camera.
![Page 14: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/14.jpg)
When an Homography can be used
• When the images are of the same planeCamera 1 Camera 2 Result
![Page 15: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/15.jpg)
When an Homography can be used
• When images taking using same camera by only rotating it
![Page 16: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/16.jpg)
Homography computation
• Using the Homography matrix H we can map point from one image to second image
• So we have : p’=Hp• P and p’ are given in homogeneous
coordinates11 12 13
21 22 23
31 32 331 1
x h h h xy h h h y
h h h
![Page 17: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/17.jpg)
Homography computation
• H is 3x3• That is 8 D.O.F• To find H we need 4 corassponding points
11 12 13 31 32' ' ' 0h x h y h h xx h yx x
21 22 23 31 32' ' ' 0h x h y h h xy h yy y
![Page 18: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/18.jpg)
Finding corresponding points
• Manually, detecting by hand similar features.
• Not accurate• Good for 2 cameras, what about 9 or more.
![Page 19: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/19.jpg)
Known solution
• Automatic Detection of known features.
• Large working volume need large objects.• very hard to detect from far distance.
![Page 20: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/20.jpg)
Features detection in wide base line
• Noise • Hidden parts• Assuming detection is possible finding the
corresponding is hard.
![Page 21: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/21.jpg)
Example of feature detection problems
![Page 22: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/22.jpg)
Goals of the calibration object
• 360 degrees view.• Robust to noise.• Accurate regardless the distance (or zoom).• Easy to find corresponding points.• Automated as possible.
![Page 23: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/23.jpg)
Solution
• Use easy to detect features (Active features).
• Use the benefits of time dimension video.• This will create a easy to detect
corresponding point list.• Find Homography using the list of points.
![Page 24: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/24.jpg)
Calibration object
• Ultra bright LEDs.
• Very bright, easy to detect.
![Page 25: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/25.jpg)
Use flickering as identifier
• features flicker in constant rate • Each feature has a different rate• The cameras filming in constant rate• The LED flicker can be found• The result a list of points in an increasing
frequency rate for each camera
![Page 26: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/26.jpg)
Detection method first stage
• Filter unnecessary noise• Use the red channel only as filter.• What about acceptable red channel filters in
RGB such as:R = ((R-B)+(R-G)).• Remove white pixels (All channels have
high intensities ).• Not good for a case a LED caused high
saturation (appears as white).
![Page 27: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/27.jpg)
Filter Example
• Red channel only ((R-B)+(R-G))
![Page 28: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/28.jpg)
Detection method second stage
• Take advantage of video camera time line• The LED is going from on to off state• Subtracting following frames (similar to
background subtraction).• Detect features pixels candidates using a
threshold.• Save detection frame number to detect
flickering rate.
![Page 29: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/29.jpg)
Detection method third stage
• So far we have points candidate and there frequencies.
• Yet some of the candidates are noise.• Use a frequency as a second filter • Most of the noises have a very short and not
consist frequency.
![Page 30: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/30.jpg)
Noise and feature frequencies
• Noise
• Feature
![Page 31: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/31.jpg)
Frequency filter
• Before
![Page 32: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/32.jpg)
Frequency filter
• After
![Page 33: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/33.jpg)
Detection method fourth stage
• Once we have the LED pixels detected we need to detect a pixel to represent it
• Local maximum, the pixel with the highest intensity level.
• Solution to different distances of camera from the features and different zoom rates.
![Page 34: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/34.jpg)
Local maximum example
• Before
![Page 35: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/35.jpg)
Local maximum example
• After
![Page 36: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/36.jpg)
Full tool example
![Page 37: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/37.jpg)
Synchronization
• Given the frame number k in the first camera find the corresponding frame in the second camera.
• Not all the cameras starts to film in the same time.
• Known solution using temporal features
![Page 38: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/38.jpg)
Temporal features
• Hard to find, not suitable for 9 cameras or more
![Page 39: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/39.jpg)
Automatic synchronization
• Each feature has a different rate• The signature is based on the gap between the
pools vibrate.• Given an index we search for the first time after
this index the pool with the lowest frequency vibrate and so on.
• Given that the polls turned on in t0,t1,t2,t3,t4,t5 the resulting signature is
• (t1-t0,t2-t1,t3-t2,t4-t3,t5-t4)
![Page 40: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/40.jpg)
Synchronization graph 1/2
![Page 41: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/41.jpg)
Synchronization graph 2/2
![Page 42: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/42.jpg)
Tool synchronization example
![Page 43: Multi video camera calibration and synchronization](https://reader035.fdocuments.us/reader035/viewer/2022062521/568167d3550346895ddd271c/html5/thumbnails/43.jpg)
The end
• Thank you!!!