Computer Vision The 2D projective plane and it’s applications HZ Ch 2. In particular: Ch 2.1-4,...

Computer Vision

The 2D projective plane and The 2D projective plane and it’s applications it’s applications

HZ Ch 2. In particular: Ch 2.1-4, 2.7,

Szelisky: Ch 2.1.1, 2.1.2Estimation:HZ: Ch 4.1-4.2.5, 4.4.4-4.8 cursorly

Richard Hartley and Andrew Zisserman, Multiple View Geometry, Cambridge University Publishers, 2nd ed. 2004

Homogeneous coordinates

0=++ cbyax ( ) ( ) 0=1x,y,a,b,c T

( ) ( ) 0≠∀,~ ka,b,cka,b,c TT

Homogeneous representation of 2D points and lines

equivalence class of vectors, any vector is representative

Set of all equivalence classes in R3(0,0,0)T forms P2

( ) ( ) 0≠∀,1,,~1,, kyxkyx TT

The point x lies on the line l if and only if


Inhomogeneous coordinates ( )Tyx,

( )T321 ,, xxx but only 2DOF

Note that scale is unimportant for incidence relation

0=xlT

X

Y

Z


X

Y

Z

s

s 0

X∞

X

Y

0

X∞ =

• Perspective imaging models 2d projective space

• Each 3D ray is a point in P2 : homogeneous coords.

• Ideal points

• P2 is R2 plus a “line at infinity” l∞

The 2D projective plane

Projective point

l∞

x

yX

Y1Z= Inhomogeneou

s equivalent

Lines

HZ • Ideal line ~ the plane parallel to the image

A

B

C

l =X=lTX = XTl = AX + BY + CZ = 0

l∞ =

0

0

1

• Projective line ~ a plane through the origin

For any 2d projective property, a dual property holds when the role of points and lines are interchanged.

Duality:

X

Y

0

X∞ =

l1 l2X =X1 X2=l

The line joining two points The point joining two lines

X

Y

Z

X

l“line at infinity

”

Points from lines and vice-versa

l'×l=x

Intersections of lines

The intersection of two lines and is l l'

Line joining two points

The line through two points and is x'×x=lx x'

Example

1=x

1=y

1

)1-,1,0( y

x

1

)1-,0,1( y

x

Note:[ ] 'xx='x×x ×

with

0-

0-

-0

x

xy

xz

yz

Ideal points and the line at infinity

T0,,l'l ab

Intersections of parallel lines

( ) ( )TTand ',,=l' ,,=l cbacba

Example

1=x 2=x

Ideal points ( )T0,, 21 xx

Line at infinity ( )T1,0,0=l∞

∞22 l∪= RP

tangent vector

normal direction

-ab,( )ba,

Note that in P2 there is no distinction

between ideal points and others

Conics

Curve described by 2nd-degree equation in the plane

0=+++++ 22 feydxcybxyax

0=+++++ 233231

2221

21 fxxexxdxcxxbxax

3

2

3

1 , xxyx

xx or homogenized

0=xx CT

or in matrix form

fed

ecb

dba

2/2/

2/2/

2/2/

Cwith

{ }fedcba :::::5DOF:

Five points define a conic

For each point the conic passes through

0=+++++ 22 feydxcyybxax iiiiii

or

( ) 0=.1,,,,, 22 ciiiiii yxyyxx ( )Tfedcba ,,,,,=c

0

1

1

1

1

1

552555

25

442444

24

332333

23

222222

22

112111

21

c

yxyyxx

yxyyxx

yxyyxx

yxyyxx

yxyyxx

stacking constraints yields

Tangent lines to conics

The line l tangent to C at point x on C is given by l=Cx

lx

C

Dual conics

0=ll *CTA line tangent to the conic C satisfies

Dual conics = line conics = conic envelopes

1-* CC In general (C full rank):

Degenerate conics

A conic is degenerate if matrix C is not of full rank

TT ml+lm=C

e.g. two lines (rank 2)

e.g. repeated line (rank 1)

Tll=C

l

l

m

Degenerate line conics: 2 points (rank 2), double point (rank1)

( ) CC ≠**Note that for degenerate conics

Conics

•Conic: Conic: – Euclidean geometry: hyperbola, ellipse, parabola & degenerateEuclidean geometry: hyperbola, ellipse, parabola & degenerate– Projective geometry: equivalent under projective transformProjective geometry: equivalent under projective transform– Defined by 5 pointsDefined by 5 points

•Tangent lineTangent line

•Dual conic C*Dual conic C*

0

022

xx C

feydxcybxyaxT

fed

ecb

dba

C

2/2/

2/2/

2/2/

xl C

0* ll CT

inhomogeneous

homogeneous

Projective transformations

• Homographies, collineations, projectivitiesHomographies, collineations, projectivities

• 3x3 nonsingular H3x3 nonsingular H

3

2

1

333231

232221

131211

3

2

1

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

l0= Hà Tl

xTl = 0 x0Tl0= 0

maps P2 to P2

8 degrees of freedomdetermined by 4 corresponding points

x0= Hx• Transforming Lines?subspaces preserved

xTHTl0= 0substitution

dual transformation

Planar Projective Warping

HZA novel view rendered via

four points with known structurexi

0= Hxii = 1. . .4

xi xi0

Planar Projective Warping

HZOriginal Top-down Facing right

Artifacts are apparent where planarity is violated...

2d Homographies

2 images of a plane

2 images from the same viewpoint (Perspectivity)

Panoramic imagingAppl: Quicktime VR, robot navigation etc.

Homographies of the world, unite!

Image mosaics are stitched by homographies

HZ

The line at infinity

l

1

0

0

1t

0ll

TT

TT

A

AH A

The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity

Note: But points on l can be rearranged to new points on l

Affine properties from images

Projection(Imaging)

RectificationPost-processing

Affine rectification

v1 v2

l1

l2 l4

l3

l∞21 vvl

211 llv 432 llv

APA

lll

HH

321

010

001

0,l 3321 llll T

Point transformation for Aff Rect:

Exercise: Verify

T321 lll

TT ]1,0,0[321 lllPAH

GroupGroup TransformationTransformation InvariantsInvariants DistortionDistortion

ProjectiveProjective

8 DOF8 DOF

• Cross ratioCross ratio

• IntersectionIntersection

• TangencyTangency

AffineAffine

6 DOF6 DOF

• ParallelismParallelism

• Relative dist in 1dRelative dist in 1d

• Line at infinityLine at infinity

MetricMetric

4 DOF4 DOF

• Relative distancesRelative distances

• AnglesAngles

• Dual conicDual conic

EuclideanEuclidean

3 DOF3 DOF

• LengthsLengths

• AreasAreas

Geometric strata: 2d overview

1TS

sRH

O

t

1TA

AH

O

t

1TE

RH

O

t

v

AH TP v

t

C*

2 dofl

2 dof

l

C*

Parameter estimation in geometric transforms

•2D homography2D homographyGiven a set of (xGiven a set of (xii,x,xii’), compute H (x’), compute H (xii’=Hx’=Hxii))

•3D to 2D camera projection3D to 2D camera projectionGiven a set of (XGiven a set of (Xii,x,xii), compute P (x), compute P (xii=PX=PXii))

•Fundamental matrixFundamental matrixGiven a set of (xGiven a set of (xii,x,xii’), compute F ’), compute F (x(xii’’TTFxFxii=0)=0)

•Trifocal tensor Trifocal tensor Given a set of (xGiven a set of (xii,x,xii’,x’,xii”), compute T”), compute T

cs499

Useful in

Grad research

Math tools 1:Solving Linear Systems

• If If mm == nn ( (AA is a square matrix), then we can obtain is a square matrix), then we can obtain the solution by simple inversion:the solution by simple inversion:

• If If mm >> nn, then the system is , then the system is over-constrainedover-constrained and and AA is not invertible is not invertible – Use Matlab “\” to obtain Use Matlab “\” to obtain least-squares solutionleast-squares solution xx == A\b A\b to to Ax Ax

=b =b internally Matlab uses QR-factorization (cmput418/340) to solve this.internally Matlab uses QR-factorization (cmput418/340) to solve this. – Can also write this using pseudoinverse Can also write this using pseudoinverse AA+ + == ((AATTAA))-1-1AATT to obtain to obtain

least-squares solutionleast-squares solution xx == A A++bb

Fitting Lines

• A 2-D point A 2-D point xx = ( = (xx, , yy)) is on a line with slope is on a line with slope

mm and intercept and intercept bb if and only if if and only if yy = = mxmx + + bb • Equivalently,Equivalently,

• So the line defined by two points So the line defined by two points xx11, , xx22 is the is the

solution to the following system of equations:solution to the following system of equations:

Fitting Lines

• With more than two points, there is no guarantee With more than two points, there is no guarantee that they will all be on the same linethat they will all be on the same line

• Least-squares solution obtained from Least-squares solution obtained from pseudoinverse is line that is “closest” to all of the pseudoinverse is line that is “closest” to all of the pointspoints

courtesy ofVanderbilt U.

Example: Fitting a Line

• Suppose we have points Suppose we have points (2, 1)(2, 1), , (5, 2)(5, 2), , (7, (7, 3)3), and , and (8, 3)(8, 3)

• Then Then

and and x x == A A++b b == (0.3571, 0.2857)(0.3571, 0.2857)TT

Matlab: Matlab: x x == A\b A\b

Example: Fitting a Line

Homogeneous Systems of Equations

• Suppose we want to solve Suppose we want to solve AA x = 0x = 0• There is a trivial solution There is a trivial solution x = 0x = 0, but we don’t want , but we don’t want

this. For what other values of this. For what other values of xx is is AA xx close to close to 00??• This is satisfied by computing the This is satisfied by computing the singular value singular value

decompositiondecomposition (SVD) (SVD) A A == UDV UDVTT (a non- (a non-negative diagonal matrix between two orthogonal negative diagonal matrix between two orthogonal matrices) and taking matrices) and taking xx as the last column of as the last column of VV– Note that Matlab returns Note that Matlab returns [U, D, V] = svd(A)[U, D, V] = svd(A)

Line-Fitting as a Homogeneous System

• A 2-D homogeneous point A 2-D homogeneous point xx = ( = (xx, , yy, 1), 1)TT is on is on

the line the line ll = ( = (aa, , bb, , cc))TT only when only when

axax + + byby + + cc = 0 = 0• We can write this equation with a dot product: We can write this equation with a dot product:

x•x• ll = 0 = 0,, and hence the following system is and hence the following system is

implied for multiple points implied for multiple points xx11,, x x22, ..., , ..., xxnn::

Example: Homogeneous Line-Fitting

• Again we have 4 points, but now in homogeneous form: Again we have 4 points, but now in homogeneous form:

(2, 1, 1)(2, 1, 1), , (5, 2, 1)(5, 2, 1), , (7, 3, 1)(7, 3, 1), and , and (8, 3, 1)(8, 3, 1)• Our system is:Our system is:

• Taking the SVD of Taking the SVD of AA, we get:, we get:compare to x = (0.3571, 0.2857)T

Parameter estimation in geometric transforms

•2D homography2D homographyGiven a set of (xGiven a set of (xii,x,xii’), compute H (x’), compute H (xii’=Hx’=Hxii))

•3D to 2D camera projection3D to 2D camera projectionGiven a set of (XGiven a set of (Xii,x,xii), compute P (x), compute P (xii=PX=PXii))

•Fundamental matrixFundamental matrixGiven a set of (xGiven a set of (xii,x,xii’), compute F ’), compute F (x(xii’’TTFxFxii=0)=0)

•Trifocal tensor Trifocal tensor Given a set of (xGiven a set of (xii,x,xii’,x’,xii”), compute T”), compute T

cs499

Useful in

Grad research

Estimating Homography Hgiven image points x

HZA novel view rendered via

four points with known structurexi

0= Hxii = 1. . .4

xi xi0

Number of measurements required

• At least as many independent equations as degrees of At least as many independent equations as degrees of freedom requiredfreedom required

• Example: Example:

Hxx'

1'

λ

333231

232221

131211

y

x

hhh

hhh

hhh

w

y

x

2 independent equations / point8 degrees of freedom

4x2≥8

Approximate solutions

•Minimal solutionMinimal solution4 points yield an exact solution for H4 points yield an exact solution for H

•More pointsMore points– No exact solution, because measurements are inexact No exact solution, because measurements are inexact

(“noise”)(“noise”)– Search for “best” according to some cost functionSearch for “best” according to some cost function– Algebraic or geometric/statistical costAlgebraic or geometric/statistical cost

Many ways to solve:

Different Cost functions => differencesDifferent Cost functions => differencesin solution in solution

•Algebraic distanceAlgebraic distance

•Geometric distanceGeometric distance

•Reprojection errorReprojection error

•ComparisonComparison

•Geometric interpretationGeometric interpretation

Gold Standard algorithm

•Cost function that is optimal for some Cost function that is optimal for some assumptionsassumptions

•Computational algorithm that minimizes it is Computational algorithm that minimizes it is called “Gold Standard” algorithm called “Gold Standard” algorithm

•Other algorithms can then be compared to itOther algorithms can then be compared to it

Estimating H: The Direct Linear Transformation (DLT) Algorithm

• xxi i =H=HXXii is an equation involving homogeneous is an equation involving homogeneous

vectors, so vectors, so HHXXii and and xxi i need only be in the same need only be in the same

direction, not strictly equaldirection, not strictly equal

• We can specify “same directionality” by using a We can specify “same directionality” by using a cross product formulation:cross product formulation:

0Hxx ii

Direct Linear Transformation(DLT)

ii Hxx 0Hxx ii

i

i

i

i

xh

xh

xh

Hx3

2

1

T

T

T

iiii

iiii

iiii

ii

yx

xw

wy

xhxh

xhxh

xhxh

Hxx12

31

23

TT

TT

TT

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

Tiiii wyx ,,x

0hA i


• Equations are linear in Equations are linear in hh

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

0AAA 321 iiiiii wyx

0hA i

• Only 2 out of 3 are linearly independent Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)(indeed, 2 eq/pt)

0

h

h

h

x0x

xx0

3

2

1

TTT

TTT

iiii

iiii

xw

yw

(only drop third row if wi’≠0)• Holds for any homogeneous representation, Holds for any homogeneous representation,

e.g. (e.g. (xxii’,’,yyii’,1)’,1)


•Solving for Solving for HH

0Ah 0h

A

A

A

A

4

3

2

1

size A is 8x9 or 12x9, but rank 8

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interestpick for example the one with 1h


•Over-determined solutionOver-determined solution

No exact solution because of inexact measurementi.e. “noise”

0Ah 0h

A

A

A

n

2

1

Find approximate solution- Additional constraint needed to avoid 0, e.g.

- not possible, so minimize

1h Ah0Ah

DLT algorithm

ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h (reshape)

Inhomogeneous solution

'

'h~

''000'''

'''''000

ii

ii

iiiiiiiiii

iiiiiiiiii

xw

yw

xyxxwwwywx

yyyxwwwywx

Since h can only be computed up to scale, pick hj=1, e.g. h9=1, and solve for 8-vector

h~

Solve using Gaussian elimination (4 points) or using linear least-squares (more than 4 points)

However, if h9=0 this approach fails also poor results if h9 close to zeroTherefore, not recommended for general homographiesNote h9=H33=0 if origin is mapped to infinity

0

1

0

0

H100Hxl 0

T

Normalizing transformations

• Since DLT is not invariant to coordinate Since DLT is not invariant to coordinate transforms, what is a good choice of transforms, what is a good choice of coordinates?coordinates?e.g.e.g.– Translate centroid to originTranslate centroid to origin– Scale to a average distance to the originScale to a average distance to the origin– Independently on both imagesIndependently on both images

2

1

norm

100

2/0

2/0

T

hhw

whwOr

Normalized DLT algorithm

ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii inormiinormi xTx~,xTx~

norm-1

norm TH~

TH

Importance of normalization

0

h

h

h

0001

1000

3

2

1

iiiiiii

iiiiiii

xyxxxyx

yyyxyyx

~102 ~102 ~102 ~102 ~104 ~104 ~10211

orders of magnitude difference!

Degenerate configurations

x4

x1

x3

x2

x4

x1

x3

x2H? H’?

x1

x3

x2

x4

0Hxx iiConstraints: i=1,2,3,4

TlxH 4* Define:

4444* xxlxxH kT

3,2,1 ,0xlxxH 4* iii

TThen,

H* is rank-1 matrix and thus not a homography

(case A) (case B)

If H* is unique solution, then no homography mapping xi→xi’(case B)

If further solution H exist, then also αH*+βH (case A) (2-D null-space in stead of 1-D null-space)

•First 3D proj geomFirst 3D proj geom

•Then review and more on camera modelsThen review and more on camera models

•Then following P estimationThen following P estimation

• Projection equationProjection equation

xxii=P=PiiXX

• Resection:Resection:– xxii,X P,X Pii

A 3D Vision Problem: Multi-view geometry - resection

Given image points and 3D points calculate camera projection matrix.

Estimating camera matrix P

•Given a number of correspondences between 3-Given a number of correspondences between 3-

D points and their 2-D image projections D points and their 2-D image projections XXii x xii, we would like to determine the , we would like to determine the camera camera

projection matrixprojection matrix PP such that such that xxi i == PXPXii for for

all all ii

A Calibration Target

courtesy of B. Wilburn

XZ

Y

Xi

xi

Estimating P: The Direct Linear Transformation (DLT) Algorithm

• xxi i == PXPXii is an equation involving homogeneous is an equation involving homogeneous

vectors, so vectors, so PXPXii and and xxi i need only be in the same need only be in the same

direction, not strictly equaldirection, not strictly equal

• We can specify “same directionality” by using a We can specify “same directionality” by using a cross product formulation:cross product formulation:

DLT Camera Matrix Estimation: Preliminaries

•Let the image point Let the image point xxii = ( = (xxii, , yyii, , wwii))TT

(remember that (remember that XXii has 4 elements) has 4 elements)

•Denoting the Denoting the jjth row of th row of PP by by ppjTjT (a 4-element (a 4-element row vector), we have: row vector), we have:

DLT Camera Matrix Estimation: Step 1

•Then by the definition of the cross product, Then by the definition of the cross product,

xxi i PXPXii is: is:


•The dot product commutes, so The dot product commutes, so ppjTjT XXii ==

XXTTii ppjj, and we can rewrite the preceding as:, and we can rewrite the preceding as:


• Collecting terms, this can be rewritten as a Collecting terms, this can be rewritten as a matrix product:matrix product:

where where 00TT = (0, 0, 0, 0) = (0, 0, 0, 0). This is a . This is a 3 x 12 matrix times a 12-element column 3 x 12 matrix times a 12-element column vector vector pp = ( = (pp11TT, , pp22TT, , pp33TT))TT

What We Just Did


• There are only two linearly independent rows here There are only two linearly independent rows here – The third row is obtained by adding The third row is obtained by adding xxii times the first row to times the first row to yyii times the times the

second and scaling the sum by second and scaling the sum by -1/-1/wwii


• So we can eliminate one row to obtain the So we can eliminate one row to obtain the

following linear matrix equation for the following linear matrix equation for the iith pair th pair of corresponding points:of corresponding points:

• Write this as Write this as AAii pp = = 00


• Remember that there are 11 unknowns which Remember that there are 11 unknowns which generate the 3 x 4 homogeneous matrix generate the 3 x 4 homogeneous matrix PP

(represented in vector form by (represented in vector form by pp))• Each point correspondence yields 2 equations Each point correspondence yields 2 equations

(the two row of (the two row of AAii)) We need at least 5 ½ point correspondences to We need at least 5 ½ point correspondences to

solve for solve for pp• Stack Stack AAii to get homogeneous linear system to get homogeneous linear system AA p p

= 0= 0

Experiment:

short and long focal length

Radial Distortion

Radial Distortion

Correction of distortion

Choice of the distortion function and center

Computing the parameters of the distortion function(i) Minimize with additional unknowns(ii) Straighten lines(iii) …

Radial Distortion

Computer Vision The 2D projective plane and it’s applications HZ Ch 2. In particular: Ch 2.1-4,...

Documents

Transcript of Computer Vision The 2D projective plane and it’s applications HZ Ch 2. In particular: Ch 2.1-4,...