3D computer vision
Transcript of 3D computer vision
3D computer vision techniques v.4b2 1
3D computer vision techniques
KH Wong
3D computer vision techniques v.4b2 2
Seminar Title: 3D computer vision techniques. Abstract
In this talk, the ideas of obtaining 3D information of objects (or called 3D reconstruction) using different techniques are discussed. Currently, the most popular one is the image based method that uses 2D cameras for 3D reconstruction; in particular reconstruction based on one-image, two-image and multiple-image are discussed. Moreover, batch and sequential treatments of input data are studied. I will also talk about novel techniques, such as using multiple cameras and laser based methods to obtain 3D information. And I will discuss how 3D computer vision is used in film and game production. Finally naked-eye 3D display technologies will be mentioned.
3D computer vision techniques v.4b2 3
Overview (part1) Introduction From 2D to 3D
Camera systems/calibration Feature extraction/correspondence Reconstruction algorithms
2 views, 3 views , N views Real-time algorithms/Kalman filter
Previous projects Virtual viewer/ Projector camera systems Keystone correction
Novel setups Multiple cameras/ Camera array
Obtain 3D directly Structured light Laser approach Kinect approach Photometric stereo
3D computer vision techniques v.4b2 4
Overview (part 2) Applications
Photos from tourists (photo tourism) http://phototour.cs.washington.edu/
3D displays Possible future research
Classification based on 3D information Content search 3D based on 3D keys Merging with sound information
3D computer vision techniques v.4b2 5
Motivation We live in a 3D world We see 2D images but perceive the world in
3D Intelligent robot should have this 3D
reconstruction capability
3D computer vision techniques v.4b2 6
How to obtain 3D information? Cameras-2D Range sensors-3D
3D computer vision techniques v.4b2 7
Challenges Obtain 3D information for tasks in a 3D world.
2D-to-3D reconstruction from a camera 3D directly— laser range sensor, kinect sensor
Novel sensors Camera array/ multiple camera One pixel camera light field camera
3D computer vision techniques v.4b2 8
2D-to-3D reconstruction (feature based method) Camera (perspective projection) Features-extraction and correspondences Methods
One-image method Two-image (Stereo) method Three-image method N-image method
Bundle adjustment Kalman filter
3D computer vision techniques v.4b2 9
Camera: 3D to 2D projection
Perspective model u=F*X/Z (nonlinear relation) v=F*Y/Z
FZ
Y
v
World center
F
Thin lensor a pin hole
Virtual Screenor CCD sensor
RealScreenOr CCD sensor
Pinhole Camera
http://upload.wikimedia.org/wikipedia/en/8/81/Pinhole-camera.png
3D computer vision techniques v.4b2 10
Perspective Projective
Model M at t=1
c (Image center, ox,oy)
F=focal length
image
Oc=(0,0,0) (Camera center)
Xc-axis
Zc-axis
Yc-axis
v-axis
u-axis
X,Y,Z
(u,v)
(0,0) of image plane
Camera Coordinates.
WorldCoordinates
YwZw
Xw Rc,Tc
Principal axis
3D computer vision techniques v.4b2 11
In paintings Western Fresco by Raphael, 1510 -
1511, Stanza della Signatura, Vatican Palace, Rome.
Chinese 《富春山居圖》是元朝畫家黃公望的作品,創作於
1347年至 1350年 Dwelling in the
Fuchun Mountains (富春山居圖 ) by Huang Gongwang (1269–1354)
http://www.es.flinders.edu.au/~mattom/science+society/lectures/illustrations/lecture17/schoolathens.htmlhttp://jsl641124.blog.163.com/blog/static/17702514320115219508530/
3D computer vision techniques v.4b2 12
Feature correspondences--Camera moved, find correspondences for neighboring images--We can use feature to identify the motions of projected 3D features in 2D.
Image at t=t0
(or left image)Image at t=t0+dt (or right image)
Area a
3D computer vision techniques v.4b2 13
Demo Youtube Movie http://www.youtube.com/watch?v
=azl-DGK6e1U
3D computer vision techniques v.4b2 14
One-image 2D-to-3D reconstruction
3D computer vision techniques v.4b2 15
One image 2D-to-3D reconstruction method Difficult and with ambiguity
http://ai.stanford.edu/~asaxena/reconstruction3d/
3D computer vision techniques v.4b2 16
One image 2D-to-3D Using prior knowledge (e.g. face)
http://www.wisdom.weizmann.ac.il/~ronen/papers/Hassner Basri - Example Based 3D Reconstruction from Single 2D Images.pdf
3D computer vision techniques v.4b2 17
Two-image 2D-to-3D reconstruction
3D computer vision techniques v.4b2 18
Two-image 2D-to-3D reconstruction method: stereo vision
Objectives: Basic idea of stereo vision Stereo reconstruction by epipolar geometry
Stereo camera pair calibration (find Fundamental matrix F)
Construct the 3D (graphic) model from 2 images
Graphic model
Inside a computer
3D computer vision techniques v.4b2 19
X’l X’r
FocalLengthf
ObjectPx(x,y,z)
z
Left camera center (reference point) Horizontal
Disparity=xL-xR
b (Baseline)
Left CameraPrincipleaxis
Right CameraPrincipleaxis
LeftImage plane
RightImage plane
,
( )
elimate (
'l
'r
' 'l r
x x=z fx - b x =
z ff bx z =x - x )
By similar triangle,w.r.t left camera lens center
By similar triangle,w.r.t right camera lens center
if camera motion is pure translation : Triangular calculation
One major problem is to locate x’l and x’r The correspondence problem
3D computer vision techniques v.4b2 20
If camera motion is NOT pure translation : Use Epipolar Geometry
X
O2O1
right Frame Plane-2 2
left Frame Plane-1 1
e1e2
Left epipolar lineRight epipolar line
R,T
(x1,y1)
(x2,y2)
Left side is the reference
Focal length=f1
Focal length=f2
Base line=||T||
Plane-3 3
Perpendicular to TX2 or TX1
Right_image_pointT*E*left_image_point=0
3D computer vision techniques v.4b2 21
Method: 8-point algorithmhttp://www.cs.manchester.ac.uk/ugt/COMP37111/papers/Hartley.pdf
Find 8 point corresponded ( Map 8 Right_image_points to left_image_point
Solve the epeiolar formula Right_image_pointT*E*left_image_point=0 Find E. From E we can find camera R (rotation) ,T
(translation) From R,T we can find model (3D positions of the
left feature points (using left as reference)
3D computer vision techniques v.4b2 22
An example of stereo reconstruction An example Short-Baseline
Stereo Systems for Mobile Devices
http://www.lelaps.de/videos.html#SQx5vU8BA-Mhttp://www.lelaps.de/projects/stmobile.html
3D computer vision techniques v.4b2 23
Stereo-based Free-space Estimation Another example
http://www.lelaps.de/videos.html#VrKBNtAN03ohttp://www.lelaps.de/projects/freespace.html
3D computer vision techniques v.4b2 24
Three-image 2D-to-3D reconstruction
3D computer vision techniques v.4b2 25
Three-image 2D-to-3D reconstruction method More robust using 3 views It contains 3 epipolar
relations Stereo1: view1,2 , Stereo2: view2,3, Stereo3 :view 3,1.
Combine 3 epipolar geometry information together.
Similar to the algorithm in epipolar geometry (apply 3 times)
http://www.cs.unc.edu/~marc/tutorial/node45.html
M=3-D model pointM, m’, m” are image pointsC,C’,C” are camera centers
3D computer vision techniques v.4b2 26
Example of 3-image reconstruction Example
LIBVISO: Feature Matching for Visual Odometry http://www.youtube.com/watch?v=DPLh6MoxPAk
3D computer vision techniques v.4b2 27
N-image
2D-to-3D reconstruction(batched method: order of
images can be random )
3D computer vision techniques v.4b2 28
N-image 2D-to-3D reconstruction method Bundle adjustment approach
Guess iteratively the solution for 3D to explain the measurements of feature points in all images
Math: Q(u,v)=g(X), g is nonlinear (projection) because u=fX/Z v=fY/Z, f=focal length Given Q (image measurement) , we want to find X=(X,Y,Z)i
from image points (u,v)i of all N model points (i=1,,,N), g is the projection formulas
A typical non linear optimization problem, Gauss-Newton for non linear optimization method is used.
3D computer vision techniques v.4b2 29
Batched method: order of images can be random From measurement [u,v]I find X
…
Camera motion
Imaget=1
Imaget=2 Image
t=3Imaget=m
…
v1
v2v3
vm
X
[u,v]2
[u,v]1
[u,v]3
[u,v]m O1
O2
O3
Om
R2,T2R3,T3
Rm,Tm
R1,T1
3D computer vision techniques v.4b2 30
Example Bundle adjustment reconstruction
http://www.cse.cuhk.edu.hk/%7Ekhwong/demo/canyon2b2.mpg
3D computer vision techniques v.4b2 31
N-image
2D-to-3D reconstruction(Sequential method: order of images are used like in a move )
3D computer vision techniques v.4b2 32
Sequential method: order of images are used like in a move From measurement [u,v]I find X
…
Camera motion
Imaget=1
Imaget=2 Image
t=3Imaget=m
…
v1
v2v3
vm
X
[u,v]2
[u,v]1
[u,v]3
[u,v]m O1
O2
O3
Om
R2,T2R3,T3
Rm,Tm
R1,T1
3D computer vision techniques v.4b2 33
Kalman Filter
33pictures by Ko Hoi Fung
3D computer vision techniques v.4b2 34
Kalman filter example
34
t = 0:t = 0:Position = x0Position = x0Velocity = v0Velocity = v0
t’ = 1:t’ = 1:Position = x1’Position = x1’Velocity = v1’Velocity = v1’x1’ = v0 * t + x0x1’ = v0 * t + x0
t = 1:t = 1:Position = x1Position = x1
PredictionPrediction
UpdateUpdate
3D computer vision techniques v.4b2 35
Example Hernan Badino and Takeo
Kanade:"A Head-Wearable Short-Baseline Stereo System for the Simultaneous Estimation of Structure and Motion".IAPR Conference on Machine Vision Applications (MVA), Nara, Japan, June
2011 http://www.youtube.com/watch?v=SQx5vU8BA-M
3D computer vision techniques v.4b2 36
Novel sensors : Camera array/ Multiple camera systems Camera array/ multiple camera: High
Performance Imaging - Using Large Camera Array
http://www.youtube.com/watch?v=0W_1Ce2lTBohttp://graphics.stanford.edu/papers/CameraArray/
3D computer vision techniques v.4b2 37
The Self-Reconfigurable Camera Array
http://chenlab.ece.cornell.edu/projects/MobileCamArray/
Demo moviehttp://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/train.movhttp://chenlab.ece.cornell.edu/projects/MobileCamArray/videos/self_reconfiguration.mov
Each camera
3D computer vision techniques v.4b2 38
Applications
3D computer vision techniques v.4b2 39
Photo tourism
http://phototour.cs.washington.edu/
3D computer vision techniques v.4b2 40
Projector-camera system
Application of computer vision
3D computer vision techniques v.4b2 41
A Projector-Camera system
3D computer vision techniques v.4b2 42
Projector-Camera calibration
http://www.youtube.com/watch?v=YHhQSglmuqY&feature=channel_page
3D computer vision techniques v.4b2 43
Our setup
3D computer vision techniques v.4b2 44
Calibration procedure
3D computer vision techniques v.4b2 45
Quadrangle tracking
3D computer vision techniques v.4b2 46
Experiments
3D computer vision techniques v.4b2 47
Projection result
3D computer vision techniques v.4b2 48
Results
3D computer vision techniques v.4b2 49
Hand held direct manipulation 3D Display
http://www.youtube.com/watch?v=vVW9QXuKfoQ&feature=relmfu
3D computer vision techniques v.4b2 50
Keystone correction
Configuration
3D computer vision techniques v.4b2 51
Aim of this work Desired Results
Keystoned projection Corrected projection
3D computer vision techniques v.4b2 52
Overview Three major modules
Projector-camera pair calibration Projection region detection and tracking Automatic keystone correction
3D projectionregion recovery
Projector-camera pair calibration
G, K
Corrected projection
Keystoned projection
Pre-warped image3D rectangle
Keystone correctionProjection region tracking
Camera frame
Flow chart
3D computer vision techniques v.4b2 53
Pre-warp projection image
Pre-warped projection image Display result
http://www.youtube.com/watch?v=y5XYdeh8Bno&list=UUfy2EumiHMeoUorMFR0woZA&index=1&feature=plcp
3D computer vision techniques v.4b2 54
Keystone correction Some real correction results
3D computer vision techniques v.4b2 55
Obtain 3D directly Laser range sensor
Time of flight Kinect
3D computer vision techniques v.4b2 56
Photometric stereo
http://www.taurusstudio.net/research/photex/ps/equation.htm
Lamertian light formula
•Given 3 or more known light source we can find the normal N•From the set of N we can approximate the surface
http://www.wisdom.weizmann.ac.il/~vision/photostereo/
3D computer vision techniques v.4b2 57
Photometric stereo using multiple cameras and multiple light sources
Demo
Dynamic Shape Capture using Multi-View Photometric Stereo SIGGRAPH 2009 http://www.youtube.com/watch?v=9hgs5zN38lk
3D computer vision techniques v.4b2 58
Multiple cameras fro human body reconstruction
Homepage://media.au.tsinghua.edu.cn
3D computer vision techniques v.4b2 59
Experimental Results
59 3D Modeling Using MVML Dome 23/5/3
3D computer vision techniques v.4b2 60
Multiple camera doom
http://www.mpi-inf.mpg.de/~yliu/
3D computer vision techniques v.4b2 61
Structured light method Calculate the shape by how the strip is
distorted.
http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2 62
Real time Virtual 3D Scanner - Structured Light Technology Demo
http://www.youtube.com/watch?v=a6pgzNUjh_s
3D computer vision techniques v.4b2 63
Time of flight laser method Send the IR-laser light to
different directions and sense how each beam is delayed.
Use the delay to calculate the distance of the object point
http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2 64
LIDAR light detection and ranging scanner
http://hodcivil.edublogs.org/2011/11/06/lidar-%E2%80%93-light-detection-and-ranging/http://commons.wikimedia.org/wiki/File:Lidar_P1270901.jpg
Leica terrestrial lidar (light detection and ranging) scanner
http://www.youtube.com/watch?v=MuwQTc8KK44
3D computer vision techniques v.4b2 65
3D Laser Scanning - Underground Mine Mapping Demo
http://www.youtube.com/watch?v=BZbvz8fePeQ
3D computer vision techniques v.4b2 66
Motion capture for film production (MOCAP)
http://upload.wikimedia.org/wikipedia/commons/7/73/MotionCapture.jpghttp://www.naturalpoint.com/optitrack/products/s250e/indepth.html
IR light emitterand camera
http://www.youtube.com/watch?v=IxJrhnynlN8
3D computer vision techniques v.4b2 67
3D body scanner
http://www.cyberware.com/products/scanners/ps.htmlhttp://www.cyberware.com/products/scanners/wbx.html
http://www.youtube.com/watch?v=86hN0x9RycM
3D computer vision techniques v.4b2 68
3-D Face capture
http://www.captivemotion.com/products/
http://www.youtube.com/watch?v=-TTR0JrocsI&feature=related
3D computer vision techniques v.4b2 69
Dimensional Imaging 4D Video Face Capture with Textures
Dimensional Imaging 4D Video Face Capture with Textures
http://www.youtube.com/watch?v=XtTN7tWaXTM&feature=related
3D computer vision techniques v.4b2 70
Kinect Another structure light
method Use dost rather than
strips
http://www.laserfocusworld.com/articles/2011/01/lasers-bring-gesture-recognition-to-the-home.html
3D computer vision techniques v.4b2 71
Kinect Hardware
3D computer vision techniques v.4b2 72
See the IR-dots emitted by KINECT
http://www.youtube.com/watch?v=-gbzXjdHfJA
http://www.youtube.com/watch?v=dTKlNGSH9Po&feature=related
3D computer vision techniques v.4b2 73
Novel sensors : light field cameraSpin off from Stanford camera array
light field camera : LYTRO camera
Be able to refocus after the picture is taken
https://www.lytro.com/camera
http://www.youtube.com/watch?v=7QV152jc3Ac
3D computer vision techniques v.4b2 74
light field camera How does it work
http://www.quora.com/Lytro/How-does-the-new-Lytro-camera-work
3D computer vision techniques v.4b2 75
3D (Volumetric) displayRendering for an Interactive 360º Light Field Display
SIGGRAPH 2007 Papers Proceedings
http://gl.ict.usc.edu/Research/3DDisplay/
http://www.youtube.com/watch?v=h6aUIS44ezE
3D computer vision techniques v.4b2 76
Occlusion-Capable Volumetric 3D Display by Cossairt,etal.
Actuality Systems, Inc
http://www.3dcgi.com/cooltech/displays/displays.htm
http://www.youtube.com/watch?v=8KaQmn2VTzs
3D computer vision techniques v.4b2 77
3D display Using a lattice with thin slits, viewer's eyes see different pixels on the screen to create 3d perception
http://www.televisions.com/tv-articles/TV-in-3D/Displaying-3D-Without-Glasses.php
3D computer vision techniques v.4b2 78
The future Content search in 3D video data bases Shot boundary detection Video data mining Video classification
3D computer vision techniques v.4b2 79
Appendix
3D computer vision techniques v.4b2 80
Essential matrix E (a 3x3 matrix) P.110[2] X1 is 3-D X in left camera (reference) system X2 is 3-D X in right camera system
RTEE
RT(i)
TT
TT TTTTT
TTTTiRTT
TTR
T
T
TT
BABxA
where0** hence
0)( , from
0)(X as same ,0)(X so
)( lar toperpendicu is plane, same theon are and0 andmatrix symmetricskew
and ][ Since )(
by sides bothmultiply ,
12
12
2222
222
12
12
BABxA
x
x
TTTT
tttt
tt
ttt
T
][
00
0][
12
13
23
3
2
1
Exercise1:Draw vectors TX2 or TX1 in the diagram
3D computer vision techniques v.4b2 81
Essential Matrix E
Right_image_pointT*E*left_image_point=0
2 1
2 2 2 1 1 1
2 1
2 1
2 12 2 2 1 1 1
2 1
2 2 1 1 12
X * * X 0
* * 0
_ _ _ _ _ tan _ ,
* * 0
* * 0
since X, ,
T
T
T
T
T
E
X Y Z E X Y Z
f fno harm to prefix some cons t termsZ Z
f fX Y Z E X Y ZZ Z
x y f E x y f
X YX Y Z and x f y fZ Z