Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.
-
Upload
quentin-singleton -
Category
Documents
-
view
223 -
download
0
Transcript of Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.
![Page 1: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/1.jpg)
Finding A Chessboard: An Introduction To Computer Vision
Martin C. Martin
![Page 2: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/2.jpg)
March 9, 2005 6
Motivation & Inspiration
• A spare time project, because I thought it would be fun
• Project: Chess against the computer, where board and your pieces are real, it’s pieces are projected
• Working so far: very robust localizing of chessboard (full 3D location and orientation)
![Page 3: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/3.jpg)
March 9, 2005 7
Requirements
• Interaction should be as natural as possible– E.g. pieces don’t have to be centered in their squares, or even
completely in them
• Should be easy to set up and give demonstrations– Little calibration as possible– Work in many lighting conditions– Although only with this board and pieces
• Camera needs to be at angle to board• Board made by my father just before he met my mother
– My brother and I learned to play on it
• I’m not a big chess fan, I just like project
![Page 4: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/4.jpg)
March 9, 2005 8
Computer Vision ’70s to ’80s: Feature Detection
• Initial Idea: Corners are unique, look for them• Compute the “cornerness” at each pixel by
adding the values of some nearby pixels, and subtracting others, in this pattern:
+ + + - - -+ + + - - -+ + + - - -- - - + + +- - - + + +- - - + + +
• Constant Image (e.g. middle of square): output = 0
• Edge between two regions (e.g. two squares side-by-side): output = 0
• Where four squares come together: output max (+ or -)
![Page 5: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/5.jpg)
March 9, 2005 9
Corner Detector In Practice
• Actually, the absolute value of the output
![Page 6: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/6.jpg)
March 9, 2005 10
Problems With Corner Detection
• Edge effects
• Strong response for some non-corners
• Easily obscured by pieces, hand
• How to link them up when many are obscured?
• Go back to something older: Find edges
![Page 7: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/7.jpg)
March 9, 2005 11
Computer Vision ’60s toEarly ’70s: Line Drawings
![Page 8: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/8.jpg)
March 9, 2005 12
Edge Finding
• Very common in early computer vision– Early computers didn’t have much power– Very early: enter lines by hand– A little later: extract line drawing from image
• Basic idea: for vertical edge, subtract pixels on left from pixels on right
• Similarly for horizontal edge
- - - + + +- - - + + +- - - + + +- - - + + +- - - + + +- - - + + +
![Page 9: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/9.jpg)
March 9, 2005 13
Edge Finding
• Need separate mask for each orientation?• No! Can compute intensity gradient from horizontal &
vertical gradients– Think of intensity as a (continuous) function of 2D position:
f(x,y)– Rate of change of intensity in direction (u, v) is– Magnitude changes as cosine of (u, v)– Magnitude maximum when (u, v) equals (∂f/∂x, ∂f/∂y)
• (∂f/∂x, ∂f/∂y) is called the gradient – Magnitude is strength of line at this point– Direction is perpendicular to line
vy
fux
f
![Page 10: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/10.jpg)
March 9, 2005 14
Gradient
![Page 11: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/11.jpg)
March 9, 2005 15
Localizing Line• How do we decide where
lines are & where they aren’t?
• One idea: threshold the magnitude– Problem: what threshold to
use? Depends on lighting, etc.
– Problem: will still get multiple pixels at each image location
![Page 12: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/12.jpg)
March 9, 2005 16
Laplacian
• Better idea: find the peak– i.e. where the 2nd derivative
crosses zero
2
2
2
2
:y
f
x
fLaplacian
• Image shows magnitude• One ridge is positive, one negative
![Page 13: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/13.jpg)
March 9, 2005 17
Gradient: Before And After Suppression
![Page 14: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/14.jpg)
March 9, 2005 18
Extracting Whole Lines
• So Far: intensity image “lineness” image• Next: “lineness” image list of lines• Need to accumulate contributions from across
image– Could be many gaps
• Want to extract position & orientation of lines– Boundaries won’t be robust, so consider lines to run
across the entire image
![Page 15: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/15.jpg)
March 9, 2005 19
Hough Transform
• Parameterize lines by angle and distance to center of image
• Discretize these and create a 2D grid covering the entire range
• Each unsuppressed pixel is part of a line– Perpendicular to the
gradient– Add it’s strength to the
bin for that line
θ
d
![Page 16: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/16.jpg)
March 9, 2005 20
Hough Transform
AngleDis
tanc
e T
o C
ente
r
![Page 17: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/17.jpg)
March 9, 2005 21
Take The 24 Biggest Lines
![Page 18: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/18.jpg)
March 9, 2005 22
Demonstration
![Page 19: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/19.jpg)
March 9, 2005 24
Coordinate Systems
• In 2D (u, v) (i.e. on the screen):– Origin at center of screen– u horizontal, increasing to the right– v vertical, increasing down– Maximum u and v determined by field of view.
• In 3D (x, y, z) (camera’s frame):– Origin at eye– Looking along z axis– x & y in directions of u & v respectively
![Page 20: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/20.jpg)
March 9, 2005 25
2D 3D
• Perspective projection– (u, v): image coordinates (2D)– (x, y, z): world coordinates (3D)– Similar Triangles
• Free to assume virtual screen at d = 1
– Relation: u = x/z, v = y/z
d
v
z
y
![Page 21: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/21.jpg)
March 9, 2005 26
2D 3D
p = (u, v) p = (x, y, z)
u = x/z, v=y/z
![Page 22: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/22.jpg)
March 9, 2005 27
Lines in 3D map to lines on screen
• Proof: the 3D line, plus the origin (eye), form a plane.
• All light rays from the 3D line to the eye are in this plane.
• The intersection of that plane and the image plane form a line
![Page 23: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/23.jpg)
March 9, 2005 28
2D 3D
p = (u, v) p = (x, y, z)
u = x/z, v=y/z
Line Line
![Page 24: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/24.jpg)
March 9, 2005 29
From 2D Lines To 3D Lines
• If a group of lines are parallel in 3D, what’s the corresponding 2D constraint?
• Equation of line in 2D: Au + Bv + C = 0• Substituting in our formula for u & v:
– Ax/z + By/z + C = 0– Ax + By + Cz = 0
• A 3D plane through the origin containing the 3D line
– Let L = (A, B, C) & p = (x, y, z). Then L•p = 0
![Page 25: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/25.jpg)
March 9, 2005 30
2D 3D
p = (u, v) p = (x, y, z)
u = x/z, v=y/z
Line Line
Line Au + Bv + C = 0
Plane
Ax + By + Cz = 0
L•p = 0
![Page 26: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/26.jpg)
March 9, 2005 31
Recovering 3D Direction
• Let’s represent the 3D line parametrically, i.e. as the set of p0+td for all t, where d is the direction.
• The 9 parallel lines on the board have the same d but different p0.
• For all t: L•(p0+td) = L•p0 + t L•d = 0• Since p0 is on the line, L•p0 = 0.• Therefore, L•d = 0, i.e. Axd + Byd + Czd = 0• That is, the point (xd, yd, zd) (which is not necessarily on
the 3D line) projects to a point on the 2D line• d is the same for all 9 lines in a group; so it must be the
common intersection point, i.e. the vanishing point– Knowing the 2D vanishing point gives us the full 3D direction!
![Page 27: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/27.jpg)
March 9, 2005 33
Vanishing Point
![Page 28: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/28.jpg)
March 9, 2005 34
Ames Room
Viewing Direction
![Page 29: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/29.jpg)
March 9, 2005 35
2D 3D
p = (u, v) p = (x, y, z)
u = x/z, v=y/z
Line Line
Line Au + Bv + C = 0
Plane
Ax + By + Cz = 0
L•p = 0
Common Intersection Parallel Lines
Vanishing Point Common Direction
![Page 30: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/30.jpg)
March 9, 2005 36
Finding The Vanishing Point
• Want to find a group of 9 2D lines with a common intersection point
• For two lines A1u+B1v+C1 = 0 and A2u+B2v+C2 = 0, intersection point is on both lines, therefore satisfies both linear equations:
• In reality, only approximately intersect at a common location
2
1
22
11
C
C
v
u
BA
BA
![Page 31: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/31.jpg)
March 9, 2005 37
Representing the Vanishing Point
• Problem: If camera is perpendicular to board, 2D lines are parallel no solution
• Problem: If camera is almost perpendicular to board, small error in line angle make for big changes in intersection location
• Euclidean distance between points is poor measure of similarity
![Page 32: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/32.jpg)
March 9, 2005 38
Desired Metric
• Sensitivity Analysis for Intersection Point:
• Of the form u’/w, v’/w. If -1 ≤ A,B,C ≤ 1, then -2 ≤ u’, v’, w ≤ 2
• So, the only way for the intersection point to be large is if w is small; the right metric is roughly 1/w
• A representation with that property is…
1221
2112
1221
2112 ,BABA
CACAv
BABA
CBCBu
![Page 33: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/33.jpg)
March 9, 2005 39
Homogeneous Coordinates
• Introduced by August Ferdinand Möbius• An example of projective geometry• Represent a 2D point using 3 numbers• The point (u, v, w) corresponds to u/w, v/w• Formula for perspective projection means any
3D point is already the homogeneous coordinate of its 2D projection
• Multiplying by a scalar doesn’t change the point:– (cu, cv, cw) represents same 2D point as (u, v, w)
![Page 34: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/34.jpg)
March 9, 2005 40
Homogeneous Coordinates
(u, v, 1)
w=1
![Page 35: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/35.jpg)
March 9, 2005 41
Project Intersection Point Onto Unit Sphere
![Page 36: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/36.jpg)
March 9, 2005 42
Lines Become “Great Circles”
![Page 37: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/37.jpg)
March 9, 2005 43
Clustering: Computer Vision In The Nineties
• 60s and 70s: Promise of human equivalence right around the corner
• 80s: Backlash against AI– Like an “internet startup” now
• 90s: Extensions of existing engineering techniques– Applied statistics: Bayesian Networks– Control Theory: Reinforcement Learning
![Page 38: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/38.jpg)
March 9, 2005 44
K Means
![Page 39: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/39.jpg)
March 9, 2005 51
2D to 3D: Distance• How do we get the distance to the board? From the
spacing between lines.• While 3D lines map to 2D lines, points equally spaced
along a 3D line AREN’T equally spaced in 2D:
![Page 40: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/40.jpg)
March 9, 2005 52
Distance To Board
• Let c = (0, 0, zc) be the point were the z axis hits the board
• For each line in group 1, we find the 3D perpendicular distance from c to the line, along d2
• These should be equally spaced in 3D
• Find the common difference, like Millikan oil drop expr.
![Page 41: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/41.jpg)
March 9, 2005 53
Distance To Board
• Let Li = (Ai, Bi, Ci) be the ith line in group 1.• We want to find ti such that the point c + ti d2
is on Li, i.e.• Li•(c + ti d2) = 0• Li•c + ti Li•d2 = 0• ti = - Li•c / Li•d2 = - Cizc / Li•d2
• zc is the only unknown, so put that on the left• Let si = ti/zc = -Ci / Li•d2
• If we choose our 3D units so that the squares are 1x1, then the common difference is 1/zc
![Page 42: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/42.jpg)
March 9, 2005 54
Distance To Board
• So, given the si, we need to find t0 and zc such that:
• si = ti/zc = (i + t0) / zc, i = 0…8
• But, we have outliers & occasional omission
• So we use robust estimation
![Page 43: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/43.jpg)
March 9, 2005 55
Robust Estimation
• Many existing parameter estimation algorithms optimize a continuous function– Sometimes there’s a closed form (e.g. MLE of center
of Gaussian is just the sample mean)– Sometimes it’s something more iterative (e.g.
Newton-Rhapson)
• However, these are usually sensitive to outliers– Data cleaning is often a big part of the analysis– The reason why decision trees (which are extreemly
robust to outliers) are the best all-round “off-the-shelf” data mining technique
![Page 44: Finding A Chessboard: An Introduction To Computer Vision Martin C. Martin.](https://reader036.fdocuments.us/reader036/viewer/2022062322/56649cdb5503460f949a6570/html5/thumbnails/44.jpg)
March 9, 2005 56
Robust Estimation
• Find distance (and offset) to maximize score:
– Involves evaluating on a fine grid– Isn’t time consuming here, since the number
of data points is small