Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis
description
Transcript of Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis
![Page 1: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/1.jpg)
September 15, 2003 ICRA2003, Taipei
Tutorial: Breakthroughs in 3-D Tutorial: Breakthroughs in 3-D Reconstruction and Motion Reconstruction and Motion
AnalysisAnalysisJana Kosecka (CS, GMU)Jana Kosecka (CS, GMU)Yi Ma (ECE, UIUC)Yi Ma (ECE, UIUC)
Shankar Sastry (EECS, Shankar Sastry (EECS, Berkeley) Berkeley)
Stefano Soatto (CS, UCLA)Stefano Soatto (CS, UCLA)Rene Vidal (Berkeley, John Rene Vidal (Berkeley, John
Hopkins)Hopkins)Omid Shakernia (EECS, Omid Shakernia (EECS,
Berkeley)Berkeley)
![Page 2: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/2.jpg)
September 15, 2003 ICRA2003, Taipei
PRIMARY REFERENCE
![Page 3: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/3.jpg)
September 15, 2003 ICRA2003, Taipei
Lecture A: Lecture A: Overview and IntroductionOverview and Introduction
Yi Yi Ma Ma
Breakthroughs in 3D Reconstruction and Motion Analysis
Perception & Decision LaboratoryDecision & Control Group, CSL
Image Formation & Processing Group, BeckmanElectrical & Computer Engineering Dept., UIUC
http://decision.csl.uiuc.edu/~yima
![Page 4: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/4.jpg)
September 15, 2003 ICRA2003, Taipei
TUTORIAL LECTURES
A. Overview and introduction (Ma) B. Preliminaries: geometry & image formation (Ma)C. Image primitives & correspondence (Kosecka)D. Two-view geometry (Kosecka) E. Uncalibrated geometry and stratification (Ma)F. Multiview recon. from points and lines (Kosecka)G. Reconstruction from scene knowledge (Ma)H. Step-by-step building of 3-D model (Kosecka)I. Landing of unmanned aerial vehicles (Kosecka)J. Multiple motion estimation (Ma)
![Page 5: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/5.jpg)
September 15, 2003 ICRA2003, Taipei
IMAGING and VISION: From 3D to 2D and then back again
GEOMETRY FOR TWO VIEWS
GEOMETRY FOR MULTIPLE VIEWS
APPLICATIONS: Image based modeling, vision based control
MULTIVIEW GEOMETRY WITH SYMMETRY
TUTORIAL LOGICAL FLOW
![Page 6: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/6.jpg)
September 15, 2003 ICRA2003, Taipei
Reconstruction from images – The Fundamental Problem
Input: Corresponding “features” in multiple perspective images.Output: Camera pose, calibration, scene structure representation.
![Page 7: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/7.jpg)
September 15, 2003 ICRA2003, Taipei
Fundamental Problem: An Anatomy of Cases
surface
curve
line
point
2 views3 views
4 views
m views
theoryalgorithm
practice
Euclideanaffine
projective
algebrageometry
optimization
![Page 8: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/8.jpg)
September 15, 2003 ICRA2003, Taipei
VISION AND GEOMETRY – An uncanny deja vu?
“The rise of projective geometry made such an overwhelmingimpression on the geometers of the first half of the nineteenthcentury that they tried to fit all geometric considerations into the projective scheme. ... The dictatorial regime of the projective idea in geometry was first successfully broken by the German astronomer and geometer Mobius, but the classical document of the democratic platform in geometry establishing the group of transformations as the ruling principle in any kind of geometry and yielding equal rights to independent consideration to each and any such group, is F. Klein's Erlangen program.”
-- Herman Weyl, Classic Groups, 1952
Synonyms: Group = Symmetry
![Page 9: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/9.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Autonomous Highway Vehicles
Image courtesy of California PATH
![Page 10: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/10.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Unmanned Aerial Vehicles (UAVs)
Courtesy of Berkeley Robotics Lab
Rate: 10Hz; Accuracy: 5cm, 4o
![Page 11: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/11.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Real-Time Virtual Object Insertion
UCLA Vision Lab
![Page 12: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/12.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Real-Time Sports Coverage
Courtesy of Princeton Video Image, Inc.
First-down line and virtual advertising
![Page 13: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/13.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Image Based Modeling and Rendering
Image courtesy of Paul Debevec
![Page 14: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/14.jpg)
September 15, 2003 ICRA2003, Taipei
APPLICATIONS – Image Alignment, Mosaicing, and Morphing
![Page 15: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/15.jpg)
September 15, 2003 ICRA2003, Taipei
GENERAL STEPS – Feature Selection and Correspondence
1. Small baselines versus large baselines2. Point features versus line features
![Page 16: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/16.jpg)
September 15, 2003 ICRA2003, Taipei
GENERAL STEPS – Structure and Motion Recovery
1. Two views versus multiple views2. Discrete versus continuous motion3. General versus planar scene4. Calibrated versus uncalibrated camera5. One motion versus multiple motions
![Page 17: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/17.jpg)
September 15, 2003 ICRA2003, Taipei
GENERAL STEPS – Image Stratification and Dense Matching
Left
Right
![Page 18: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/18.jpg)
September 15, 2003 ICRA2003, Taipei
GENERAL STEPS – 3-D Surface Model and Rendering
1. Point clouds versus surfaces (level sets)2. Random shapes versus regular structures
![Page 19: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/19.jpg)
September 15, 2003 ICRA2003, Taipei
Lecture B: Lecture B: Rigid-Body Motion and Imaging Rigid-Body Motion and Imaging
GeometryGeometryYi Yi
Ma Ma Perception & Decision LaboratoryDecision & Control Group, CSL
Image Formation & Processing Group, BeckmanElectrical & Computer Engineering Dept., UIUC
http://decision.csl.uiuc.edu/~yima
Breakthroughs in 3D Reconstruction and Motion Analysis
![Page 20: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/20.jpg)
September 15, 2003 ICRA2003, Taipei
GEOMETRIC MODELS OF IMAGE FORMATION• Pinhole camera model
SUMMARY OF NOTATION
CAMERA INTRINSIC PARAMETERS • From metric to pixel coordinates
3-D EUCLIDEAN SPACE & RIGID-BODY MOTION• Coordinates and coordinate frames• Rigid-body motion and homogeneous coordinates
OUTLINE
![Page 21: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/21.jpg)
September 15, 2003 ICRA2003, Taipei
3-D EUCLIDEAN SPACE - Cartesian Coordinate Frame
Standard base vectors:
Coordinates of a point in space:
![Page 22: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/22.jpg)
September 15, 2003 ICRA2003, Taipei
3-D EUCLIDEAN SPACE - Vectors
A “free” vector is defined by a pair of points :
Coordinates of the vector :
![Page 23: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/23.jpg)
September 15, 2003 ICRA2003, Taipei
3-D EUCLIDEAN SPACE – Inner Product and Cross Product
Inner product between two vectors:
Cross product between two vectors:
![Page 24: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/24.jpg)
September 15, 2003 ICRA2003, Taipei
RIGID-BODY MOTION – Rotation
Rotation matrix:
Coordinates are related by:
![Page 25: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/25.jpg)
September 15, 2003 ICRA2003, Taipei
RIGID-BODY MOTION – Rotation and Translation
Coordinates are related by:Velocities are related by:
![Page 26: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/26.jpg)
September 15, 2003 ICRA2003, Taipei
RIGID-BODY MOTION – Homogeneous Coordinates
3-D coordinates are related by:Homogeneous coordinates:
Homogeneous coordinates/velocities are related by:
![Page 27: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/27.jpg)
September 15, 2003 ICRA2003, Taipei
IMAGE FORMATION – Perspective Imaging
Image courtesy of C. Taylor
“The Scholar of Athens,” Raphael, 1518
![Page 28: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/28.jpg)
September 15, 2003 ICRA2003, Taipei
IMAGE FORMATION – Pinhole Camera Model
Pinhole
Frontalpinhole
![Page 29: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/29.jpg)
September 15, 2003 ICRA2003, Taipei
IMAGE FORMATION – Pinhole Camera Model
2-D coordinates
Homogeneous coordinates
![Page 30: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/30.jpg)
September 15, 2003 ICRA2003, Taipei
pixelcoordinates
Linear transformation
CAMERA PARAMETERS – Pixel Coordinates
metriccoordinates
![Page 31: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/31.jpg)
September 15, 2003 ICRA2003, Taipei
CAMERA PARAMETERS – Calibration Matrix and Camera Model
Pinhole camera Pixel coordinates
Calibration matrix(intrinsic parameters)Projection matrix
Camera model
![Page 32: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/32.jpg)
September 15, 2003 ICRA2003, Taipei
Homogeneous coordinates of a 3-D point
Homogeneous coordinates of its 2-D image
IMAGE FORMATION – Image of a Point
Projection of a 3-D point to an image plane
![Page 33: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/33.jpg)
September 15, 2003 ICRA2003, Taipei
Homogeneous representation of a 3-D line
IMAGE FORMATION – Image of a Line
Homogeneous representation of its 2-D image
Projection of a 3-D line to an image plane
![Page 34: Tutorial: Breakthroughs in 3-D Reconstruction and Motion Analysis](https://reader035.fdocuments.us/reader035/viewer/2022070420/56815f8f550346895dce91b9/html5/thumbnails/34.jpg)
September 15, 2003 ICRA2003, Taipei
. . .
1. Images are all “incident” at the corresponding features in space;2. Features in space have many types of incidence relationships;3. Features in space have many types of metric relationships.
SUMMARY OF NOTATION – Multiple Images