CSCE 5013 Computer Vision Fall 2011 Prof. John Gauch [email protected].
Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor....
Transcript of Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor....
![Page 1: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/1.jpg)
6.891Computer Vision and Applications
Prof. Trevor. Darrell• Class overview• Administrivia & Policies• Lecture 1
– Perspective projection (review)– Rigid motions (review)– Camera Calibration
Readings: Forsythe & Ponce, 1.1, 2.1, 2.2, 2.3, 3.1, 3.2
![Page 2: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/2.jpg)
Vision• What does it mean, to see? “to know what is where by looking”.
• How to discover from images what is present in the world, where things are, what actions are taking place.
from Marr, 1982
![Page 3: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/3.jpg)
Why study Computer Vision?• One can “see the future” (and avoid bad things…)!• Images and movies are everywhere; fast-growing
collection of useful applications– building representations of the 3D world from pictures– automated surveillance (who’s doing what)– movie post-processing– face finding
• Greater understanding of human vision• Various deep and attractive scientific mysteries– how does object recognition work?
![Page 4: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/4.jpg)
Why study Computer Vision?• People draw distinctions between what is seen– “Object recognition”– This could mean “is this a fish or a bicycle?”– It could mean “is this George Washington?”– It could mean “is this poisonous or not?”– It could mean “is this slippery or not?”– It could mean “will this support my weight?”– Great mystery
• How to build programs that can draw useful distinctions based on image properties.
![Page 5: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/5.jpg)
Computer vision class, fast-forward
![Page 6: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/6.jpg)
Cameras, lenses, and sensors
From Computer Vision, Forsyth and Ponce, Prentice-Hall, 2002.
•Pinhole cameras•Lenses•Projection models•Geometric camera parameters
![Page 7: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/7.jpg)
Image filtering• Review of linear systems, convolution• Bandpass filter-based image representations• Probabilistic models for images
ImageOriented, multi-scale representation
![Page 8: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/8.jpg)
From Foundations of Vision, by Brian Wandell, Sinauer Assoc., 1995
Color
![Page 9: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/9.jpg)
Models of texture
A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients J. Portilla and E. Simoncelli, International Journal of Computer Vision 40(1): 49-71, October 2000.© Kluwer Academic Publishers.
Parametric model Non-parametric model
A. Efros and W. T Freeman, Image quilting for texture synthesis and transfer, SIGGRAPH 2001
![Page 10: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/10.jpg)
Statistical classifiers
– MIT Media Lab face localization results.– Applications: database search, human machine interaction, video conferencing.
![Page 11: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/11.jpg)
Multi-view GeometryWhat are the relationships between images of point features in more than one view?
Given a point feature in one camera view, predict it’s location in a second (or third) camera?
![Page 12: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/12.jpg)
Ego-Motion / “Match-move”Where are the cameras?
Track points, estimate consistent poses…
Render synthetic objects in real world!
![Page 13: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/13.jpg)
Ego-Motion / “Match-move”
Video
See “Harts War” and other examples in Gallery of examples for Matchmove
program at www.realviz.com
![Page 14: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/14.jpg)
Structure from MotionWhat is the shape of the scene?
![Page 15: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/15.jpg)
SegmentationHow many ways can you segment six points?
(or curves)
![Page 16: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/16.jpg)
![Page 17: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/17.jpg)
![Page 18: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/18.jpg)
Segmentation• Which image components “belong together”?
• Belong together=lie on the same object• Cues
– similar colour– similar texture– not separated by contour– form a suggestive shape when assembled
![Page 19: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/19.jpg)
![Page 20: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/20.jpg)
![Page 21: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/21.jpg)
![Page 22: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/22.jpg)
TrackingFollow objects and estimate location..– radar / planes– pedestrians– cars– face features / expressions
Many ad-hoc approaches…General probabilistic formulation: model density
over time.
![Page 23: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/23.jpg)
Tracking• Use a model to predict next position and refine
using next image• Model:– simple dynamic models (second order dynamics)– kinematic models– etc.
• Face tracking and eye tracking now work rather well
![Page 24: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/24.jpg)
Articulated Models
Find most likely model consistent with observations….(and previous configuration)
![Page 25: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/25.jpg)
Articulated tracking• Constrained
optimization• Coarse-to-fine
part iteration• Propagate joint
constraints through each limb
• Real-time on Ghz pentium…
![Page 26: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/26.jpg)
Video
![Page 27: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/27.jpg)
And…• Visual Category Learning• Image Databases• Image-based Rendering• Visual Speechreading• Medical Imaging
![Page 28: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/28.jpg)
Administrivia• Syllabus• Grading• Collaboration Policy• Project
![Page 29: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/29.jpg)
![Page 30: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/30.jpg)
![Page 31: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/31.jpg)
Grading• Two take-home exams • Five problem sets with lab exercises in Matlab • No final exam • Final project
![Page 32: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/32.jpg)
Collaboration PolicyProblem sets may be discussed, but all written
work and coding must be done individually. Take-home exams may not be discussed. Individuals found submitting duplicate or substantially similar materials due to inappropriate collaboration may get an F in this class and other sanctions.
![Page 33: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/33.jpg)
ProjectThe final project may be
– An original implementation of a new or published idea – A detailed empirical evaluation of an existing
implementation of one or more methods – A paper comparing three or more papers not covered in
class, or surveying recent literature in a particular area A project proposal not longer than two pages must be
submitted and approved by April 1st.
![Page 34: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/34.jpg)
Problem Set 0• Out today, due 2/12• Matlab image exercises
– load, display images– pixel manipulation– RGB color interpolation– image warping / morphing with interp2– simple background subtraction
• All psets graded loosely: check, check-, 0.• (Outstanding solutions get extra credit.)
![Page 35: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/35.jpg)
![Page 36: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/36.jpg)
Cameras, lenses, and calibrationToday:• Camera models (review)• Projection equations (review)You should have been exposed to this material
in previous courses; this lecture is just a (quick) review.
• Calibration methods (new)
![Page 37: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/37.jpg)
7-year old’s question
Why is there no image on a white piece of paper?
![Page 38: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/38.jpg)
Virtual image, perspective projection
• Abstract camera model - box with a small hole in it
Forsyth&Ponce
![Page 39: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/39.jpg)
Images are two-dimensional patterns of brightness values.
They are formed by the projection of 3D objects.
Figure from US Navy Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.
![Page 40: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/40.jpg)
Animal eye: a looonnng time ago.
Pinhole perspective projection: Brunelleschi, XVth Century.Camera obscura: XVIth Century.
Photographic camera:Niepce, 1816.
Reproduced by permission, the American Society of Photogrammetry andRemote Sensing. A.L. Nowicki, “Stereoscopy.” Manual of Photogrammetry,Thompson, Radlinski, and Speert (eds.), third edition, 1966. Figure from US Navy
Manual of Basic Optics and Optical Instruments, prepared by Bureau of Naval Personnel. Reprinted by Dover Publications, Inc., 1969.
![Page 41: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/41.jpg)
The equation of projection
���
����
�
�
zyfyzxfx
''
''
![Page 42: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/42.jpg)
Distant objects are smaller
Forsyth&Ponce
![Page 43: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/43.jpg)
• Points go to points• Lines go to lines• Planes go to whole image or half-planes.
• Polygons go to polygons• Degenerate cases– line through focal point to point
– plane through focal point to line
Geometric properties of projection
pointslinesthe whole imageor a half-planepolygons
point
line���
����
�
�
zyfyzxfx
''
''
![Page 44: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/44.jpg)
Parallel lines meetCommon to draw film planein front of the focal point.Moving the film plane merelyscales the image.
Forsyth&Ponce
![Page 45: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/45.jpg)
Vanishing points• Each set of parallel lines (=direction) meets at a different point– The vanishing point for this direction
• Sets of parallel lines on the same plane lead to collinear vanishing points. – The line is called the horizon for that plane
![Page 46: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/46.jpg)
What if you photograph a brick wall head-on?
![Page 47: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/47.jpg)
Two-point perspective
http://www.sanford-artedventures.com/create/tech_2pt_perspective.html
![Page 48: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/48.jpg)
http://www.sanford-artedventures.com/create/tech_1pt_perspective.html
![Page 49: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/49.jpg)
http://www.sanford-artedventures.com/create/tech_2pt_perspective.html
![Page 50: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/50.jpg)
http://www.siggraph.org/education/materials/HyperGraph/viewing/view3d/perspect.htm
![Page 51: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/51.jpg)
Weak perspective• Issue– perspective effects, but not over the scale of individual objects
– collect points into a group at about the same depth, then divide each point by the depth of its group
– Adv: easy– Disadv: wrong
![Page 52: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/52.jpg)
Orthographic projection
![Page 53: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/53.jpg)
How large a pinhole?
![Page 54: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/54.jpg)
Wandell, Foundations of Vision, Sinauer, 1995
![Page 55: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/55.jpg)
Wandell, Foundations of Vision, Sinauer, 1995
![Page 56: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/56.jpg)
The reason for lenses
![Page 57: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/57.jpg)
Water glass refraction
http://data.pg2k.hd.org/_exhibits/natural-
science/cat-black-and-white-domestic-short-
hair-DSH-with-nose-in-glass-of-water-on-bedside-
table-tweaked-mono-1-AJHD.jpg
![Page 58: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/58.jpg)
The thin lens, first order optics
1z' - 1
z = 1f )1(2 �
� nRf
Forsyth&Ponce
All rays through P also pass through P’, but only for pointsat -z: “depth of field”.
![Page 59: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/59.jpg)
More accurate models of real lenses
• Finite lens thickness• Higher order approximation to• Chromatic aberration• Vignetting
)sin(�
![Page 60: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/60.jpg)
Thick lens
Forsyth&Ponce
![Page 61: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/61.jpg)
Lens systems
Forsyth&Ponce
Lens systems can be designed to correct for aberrations described by 3rd order optics
![Page 62: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/62.jpg)
Vignetting
Forsyth&Ponce
![Page 63: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/63.jpg)
Chromatic aberration(great for prisms, bad for lenses)
![Page 64: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/64.jpg)
Other (possibly annoying) phenomena
• Chromatic aberration– Light at different wavelengths follows different paths; hence, some wavelengths are defocussed
– Machines: coat the lens– Humans: live with it
• Scattering at the lens surface– Some light entering the lens system is reflected off each surface it encounters (Fresnel’s law gives details)
– Machines: coat the lens, interior– Humans: live with it (various scattering phenomena are visible in the human eye)
![Page 65: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/65.jpg)
Summary so far• Want to make images• Pinhole camera models the geometry of perspective projection
• Lenses make it work in practice• Models for lenses
– Thin lens, spherical surfaces, first order optics– Thick lens, higher-order optics, vignetting.
![Page 66: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/66.jpg)
Some background material…• Rigid motion: translation and rotation• Homogenous coordinates
![Page 67: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/67.jpg)
Translation����
�����
��
Z
Y
xA
AAA
P����
�����
��
Z
Y
xB
BBB
P
Ai
Ak
Aj
Bi
Bk
Bj
PxA
YA ZA ABO�
How does relate to ?PAPB
ABAB OPP ��
![Page 68: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/68.jpg)
Rotation����
�����
��
Z
Y
xB
BBB
P����
�����
��
Z
Y
xA
AAA
P
Ai
AkAj
PxA
YA ZA
How does relate to ?PAPB
PRP ABA
B �
![Page 69: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/69.jpg)
Find the rotation matrixProject
onto the B frame’s coordinate axes.
� �����
�����
��
Z
Y
X
AAA
AAA
kjiOP ˆˆˆ Ai
Ak
Aj
PxA
YA ZA
����
�
�
����
�
�
���������
�����
�����
�
ZABYABXAB
ZABYABXAB
ZABYABXAB
Z
Y
X
AkkAjkAikAkjAjjAijAkiAjiAii
BBB
ˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆ
����
�
�
����
�
�
���������
�����
�����
�
ZABYABXAB
ZABYABXAB
ZABYABXAB
Z
Y
X
AkkAjkAikAkjAjjAijAkiAjiAii
BBB
ˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆ
����
�
�
����
�
�
���������
�����
�����
�
ZABYABXAB
ZABYABXAB
ZABYABXAB
Z
Y
X
AkkAjkAikAkjAjjAijAkiAjiAii
BBB
ˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆ
![Page 70: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/70.jpg)
Rotation matrix
����
�
�
����
�
�
���������
�����
�����
�
ZABYABXAB
ZABYABXAB
ZABYABXAB
Z
Y
X
AkkAjkAikAkjAjjAijAkiAjiAii
BBB
ˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆthis
PRP ABA
B �implies
����
�
�
����
�
�
���������
�ABABAB
ABABAB
ABABABBA
kkjkikkjjjijkijiii
Rˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆˆ
where
![Page 71: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/71.jpg)
Translation and rotation
Let’s write
as a single matrix equation:
ABAB
AB OPRP ��
�����
�
�
�����
�
�
����
�
�
����
�
�
��������
������
�
�
�����
�
�
11000|
|
1Z
Y
X
ABB
A
Z
Y
X
AAA
ORBBB
![Page 72: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/72.jpg)
Homogenous coordinates• Add an extra coordinate and use an equivalence relation
• for 3D– equivalence relationk*(X,Y,Z,T) is the same as (X,Y,Z,T)
• Motivation– Possible to write the action of a perspective camera as a matrix
![Page 73: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/73.jpg)
Homogenous/non-homogenous transformations for a 3-d point
• From non-homogenous to homogenous coordinates: add 1 as the 4th coordinate, ie
• From homogenous to non-homogenous coordinates: divide 1st 3 coordinates by the 4th, ie
�����
�
�
�����
�
��
����
�����
�
1zyx
zyx
����
�����
��
�����
�
�
�����
�
�
zyx
TTzyx
1
![Page 74: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/74.jpg)
Homogenous/non-homogenous transformations for a 2-d point
• From non-homogenous to homogenous coordinates: add 1 as the 3rd coordinate, ie
• From homogenous to non-homogenous coordinates: divide 1st 2 coordinates by the 3rd, ie
����
�����
�����
�����
1yx
yx
�������
������
�����
�yx
zzyx
1
![Page 75: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/75.jpg)
The camera matrix, in homogenous coordinates
• Turn previous expression into HC’s– HC’s for 3D point are (X,Y,Z,T)
– HC’s for point in image are (U,V,W)
�����
�
�
�����
�
�
����
�
�
����
�
��
����
�
�
����
�
�
TZYX
fYX
fZ 0100
00100001
�������
������
�
�
����
�
�YX
YX
Zf
fZ
HC Non-HCWhat about an orthographic camera?
![Page 76: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/76.jpg)
The projection matrix for orthographic projection, homogenous coordinates
UVW
��
��������
��
���������
1 0 0 00 1 0 00 0 0 1
��
��������
��
��������
XYZT
��
��
����������
��
��
����������
�������
������
�����
��
YX
TTYX
1
HC Non-HC
![Page 77: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/77.jpg)
Camera calibrationUse the camera to tell you things about the
world:– Relationship between coordinates in the world and coordinates in the image: geometric camera calibration.
– (Relationship between intensities in the world and intensities in the image: photometric camera calibration, not covered in this course, see 6.801 or text)
![Page 78: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/78.jpg)
Intrinsic parameters
Forsyth&Ponce
zyfvzxfu
�
�Perspective projection
![Page 79: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/79.jpg)
Intrinsic parameters
zyfvzxfu
�
�But “pixels” are in some arbitrary spatial units…
![Page 80: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/80.jpg)
Intrinsic parameters
zyvzxu
�
�
�
�But “pixels” are in some arbitrary spatial units
![Page 81: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/81.jpg)
Intrinsic parameters
zyvzxu
�
�
�
�Maybe pixels are not square…
![Page 82: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/82.jpg)
Intrinsic parameters
zyvzxu
��
�
�Maybe pixels are not square
![Page 83: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/83.jpg)
Intrinsic parameters
zyvzxu
��
�
�We don’t know the origin of our camera pixel coordinates…
![Page 84: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/84.jpg)
Intrinsic parameters
0
0
vzyv
uzxu
��
��
��We don’t know the
origin of our camera pixel coordinates
![Page 85: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/85.jpg)
Intrinsic parameters
0
0
vzyv
uzxu
��
��
��May be skew between
camera pixel axes…
![Page 86: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/86.jpg)
Intrinsic parameters
0
0
)sin(
)cot( v
zyv
uzy
zxu
��
���
��
���May be skew between camera pixel axes
![Page 87: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/87.jpg)
Intrinsic parameters
0
0
)sin(
)cot(
vzyv
uzy
zxu
��
���
��
���
�����
�
�
�����
�
�
����
�
�
����
�
� ��
����
�����
�
1000
100)sin(0)cot(
11
0
0
zyx
v
u
zvu
��
���Using homogenous coordinates,we can write this as:
or:
� � PKzp ��
� 0 1 �
![Page 88: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/88.jpg)
Extrinsic parameters: translation and rotation of camera frame
WCWC
WC OPRP �� Non-homogeneous
coordinates
�����
�
�
�����
�
�
����
�
�
����
�
�
��������
������
�
�
�����
�
�
11000|
|
1Z
Y
X
WCC
W
Z
Y
X
WWW
ORCCC
Homogeneous coordinates
Block matrix form
![Page 89: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/89.jpg)
Combining extrinsic and intrinsic calibration parameters
Forsyth&Ponce
WCWC
WC OPRP ��
� � PORKzp WCC
W
�
� 1 �
� � PKzp ��
� 0 1 �
PΜzp �
� 1 �
Intrinsic
Extrinsic
![Page 90: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/90.jpg)
Other ways to write the same equation
�����
�
�
�����
�
�
���
�
����
�
��
����
�����
�
1.........
1
1 3
2
1
z
y
x
T
T
T
WWW
mmm
zvu
PMzp �
� 1�
PmPmv
PmPmu
�
�
�
�
�
��
�
��
3
2
3
1
pixel coordinatesworld coordinates
z is in the camera coordinate system, but we can solve for that, since , leading to:
zPm�
�� 31
![Page 91: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/91.jpg)
Calibration target
http://www.kinetic.bc.ca/CompVision/opti-CAL.html
![Page 92: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/92.jpg)
Camera calibration
0)(0)(
32
31
���
���
ii
ii
PmvmPmum�
�
So for each feature point, i, we have:
PmPmv
PmPmu
�
�
�
�
�
��
�
��
3
2
3
1From before, we had these equations relating image positions,u,v, to points at 3-d positions P (in homogeneous coordinates):
![Page 93: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/93.jpg)
Camera calibration
0)(0)(
32
31
���
���
ii
ii
PmvmPmum�
�
Stack all these measurements of i=1…n points
into a big matrix:
������
�
�
������
�
�
�����
�����
�
������
�
�
������
�
�
����
00
00
00
00
3
2
1111
111
����
mmm
PvPPuP
PvPPuP
Tnn
Tn
T
Tnn
TTn
TTT
TTT
![Page 94: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/94.jpg)
������
�
�
������
�
��
����
�����
�
������
�
�
������
�
�
����
00
00
00
00
3
2
1111
111
����
mmm
PvPPuP
PvPPuP
Tnn
Tn
T
Tnn
TTn
TTT
TTT
������
�
�
������
�
��
�����������������
�
�
�����������������
�
�
������
�
�
������
�
�
����������������
00
00
1000000001
1000000001
34
33
32
31
24
23
22
21
14
13
12
11
1111111111
1111111111
����
mmmmmmmmmmmm
vPvPvPvPPPuPuPuPuPPP
vPvPvPvPPPuPuPuPuPPP
nnznnynnxnnznynx
nnznnynnxnnznynx
zyxzyx
zyxzyx
Showing all the elements:
In vector form: Camera calibration
![Page 95: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/95.jpg)
We want to solve for the unit vector m (the stacked one)that minimizes 2Pm
������
�
�
������
�
��
�����������������
�
�
�����������������
�
�
������
�
�
������
�
�
����������������
00
00
1000000001
1000000001
34
33
32
31
24
23
22
21
14
13
12
11
1111111111
1111111111
����
mmmmmmmmmmmm
vPvPvPvPPPuPuPuPuPPP
vPvPvPvPPPuPuPuPuPPP
nnznnynnxnnznynx
nnznnynnxnnznynx
zyxzyx
zyxzyx
P m = 0
The minimum eigenvector of the matrix PTP gives us that(see Forsyth&Ponce, 3.1)
Camera calibration
![Page 96: Computer Vision and Applications Prof. Trevor. DarrellComputer Vision and Applications Prof. Trevor. Darrell • Class overview • Administrivia & Policies ... – similar colour](https://reader030.fdocuments.us/reader030/viewer/2022040507/5e463356f9dda01c483a4ff5/html5/thumbnails/96.jpg)
Once you have the M matrix, can recover the intrinsic and extrinsic parameters as in Forsyth&Ponce, sect. 3.2.2.
Camera calibration