Class 61 Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics...

Class 6 1

Multi-linear Systems and Invariant Theory

in the Context of Computer Vision and Graphics

Class 6: Duality and Shape Tensors

CS329Stanford University

Amnon Shashua

Class 6 2

Material We Will Cover Today

• The Point-View Duality Principle

• Multi-point Constraints of a single view

• Single-View Tensors

• Properties of Single View Tensors

• Single View Tensors under a stabilized reference plane

Class 6 3

Point-View Duality Principle

i

i

i

i

i

W

Z

Y

X

P points in 3D projective space

ii MPp points in 2D projective space

Approach: with enough points, we can “eliminate” M

Class 6 4


iii TPMTMPp 1

We are free to choose a projective basis for the 3D space, and we fix12 degrees of freedom (out of 16):

Let us choose a representation of 3D space such that the first 4 pointshave the “standard” coordinates:

0

0

0

1

1P

0

0

1

0

2P

0

1

0

0

3P

1

0

0

0

4P

each point fixes 3 degrees of freedom

Class 6 5


We can also choose a basis for the 2D (image) projective space:

0

0

0

1

0

0

1

M

0

0

1

0

0

1

0

M

0

1

0

0

1

0

0

M

1

0

0

0

1

1

1

M

Class 6 6


0

0

0

1

0

0

1

M

0

0

1

0

0

1

0

M

0

1

0

0

1

0

0

M

1

0

0

0

1

1

1

M

0

01

M

0

0

2 M

0

0

3M

4M

00

00

00

M

Class 6 7


00

00

00

M

Note: in the new basis, M is represented by 4 numbers (!)

Note: in the new basis we cannot assume that

1

y

x

p

(third coordinate may vanish), thus

z

y

x

p

Class 6 8


ii

ii

ii

i

i

i

i

i

WZ

WY

WX

W

Z

Y

X

p

00

00

00

00

00

00

....7,6,5ifor

MPMPp iiiˆˆ

M̂

/1

/1

/1

/1

)(Mnull

Class 6 9

Single-view Multilinear Constraints

Let ', ii ll be two lines coincident with ip

0iTi pl 0' i

Ti pl

MPp iiˆˆ

0ˆˆ MPl iTi

0ˆˆ' MPl iTi

Class 6 10

8,7,6,5ifor

Single-view Multilinear Constraints

0ˆ

ˆ.

ˆ

ˆ.

ˆ

488'8

5'5

88

55

M

Pl

Pl

Pl

Pl

T

T

T

T

Every 4x4 minor vanishes !

The choice of 4 rows can include 2,3,4 points (in addition to the 4 basis points)

Class 6 11

Single-view Bilinear Constraints

Choose the 4 rows such that they include only 2 points:

0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

Class 6 12


0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

We still have 4 d.o.f left, so we can setthe coordinates of

1

1

1

1

5P

Let

W

Z

Y

X

P6

x

z

l 0

z

y

x

p

y

zl

0

'

Class 6 13


0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

x

z

l 0

y

zl

0

'

1100

1010

1001

5̂P

WZ

WY

WX

P

00

00

00

6̂

0

0

0

0

0

det

6666

6666

5555

5555

WzWyZyYz

WzWxZxXz

zyyz

zxxz

Class 6 14


0

0

0

0

0

det

6666

6666

5555

5555

WzWyZyYz

WzWxZxXz

zyyz

zxxz

056 GppT

0

0

0

WYXYWXXY

WZXZXWXZ

WZYZYWWY

G

for all viewing positions !

65 , pp change with viewing position

Class 6 15


056 GppT

0

0

0

WYXYWXXY

WZXZXWXZ

WZYZYWWY

G

0332211 GGG ij

ijG 1

0)det( G

The dual fundamental matrix has additional 4 linear constraints!(compared to the fundamental matrix).

Class 6 16

056 GppT

The Dual Fundamental Matrix

5Gp is a line coincident with 6p

choose the camera projection matrix such that 06 MP

that is, the center of projection is at 6P

06 MPlT l 0ˆ6̂ MPlT

05 GppT p 05 Gp

0)det( GWhy ?

0)det( G

Since 056 GppT holds for all camera matrices,

Class 6 17

5p

5P

6P

5Gp

056 Ge56e

C

)ˆ(ˆ6556 PnullPe

(center of proj)

A “sketch” – the sketch is somewhat misleading since there are two (dual) epipoles.

Note: while C changes so doesthe image plane!

when 6PC

is the projection of

5P

5

5

5

5

66

66

66

6

6

6

6

55

55

55

/1/100

/10/10

/100/1

/1

/1

/1

/1

00

00

00

W

Z

Y

X

WZ

WY

WX

W

Z

Y

X

WZ

WY

WX

)ˆ(ˆ65 PnullP when 6)( PMnull projection of 5P

)ˆ(ˆ5665 PnullPe is the projection of 6P when 5PC

065 eGT

Class 6 18


0332211 GGG ij

ijG 1Why ?

Unlike the multiview tensors, the single-view point tensors have internallinear constraints (“synthetic” constraints) which are related to the factthe dual projection matrices are sparse.

Since 056 GppT holds for all camera matrices,

consider the camera matrices jM

0000

0000

0001

1M

1e

0000

0010

0000

2M

2e

0100

0000

0000

3M

3e

1000

1000

1000

4M

4e

Class 6 19


0332211 GGG ij

ijG 1Why ?

0jj ePM P

0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

'66

'55 ,,, llll coincident with je

holds for all choices of 65ˆ,ˆ PP !!

0jTjGee 4,...,1j

Class 6 20

6-point Single-view Indexing

056 GppT is a function of 6 points (4 basis, 2 additional)

Each view provides one constraint for G. How many views are necessary?

G is defined up to scale and has 4 linear constraints – thus there are4 parameters to solve for.

For a linear solution: 4 views are necessary.For non-linear: 3 views are necessary (3-fold ambiguity).

Once G is recovered, the function 056 GppT is view-independent

thus provides a connection between 6 image points and their corresponding3D points which does not depend on the point of view (indexing function).

Class 6 21

7-point Single-view Tensor

7'7

77

6'6

66

5'5

55

ˆ

ˆ

ˆˆ

ˆ

ˆ

det0

Pl

Pl

PlPl

Pl

Pl

T

T

T

T

T

T

12 choices to choose 4 rows which include 7 points (4 basis +3).there are 3 groups: choose 2 rows from a singlepoint and the remaining two rows one from each remaining point.

choose these two rows

choose 1 rows from here

choose 1 rows from here

4 trilinear constraints (view independent)(per group)

Class 6 22


7'7

77

6'6

66

5'5

55

ˆ

ˆ

ˆˆ

ˆ

ˆ

det0

Pl

Pl

PlPl

Pl

Pl

T

T

T

T

T

T

0765 jk

ikji Tllp

6l is some line incident with 6p7l is some line incident with 7p

jkiT is a trilinear function of 765 ,, PPP alone.

Class 6 23

Since holds for all camera matrices,

consider the camera matrices jM

0000

0000

0001

1M

1e

0000

0010

0000

2M

2e

0100

0000

0000

3M

3e

1000

1000

1000

4M

4e


The “synthetic” constraints for the dual trilinear tensor:

0765 jk

ikji Tllp

0jj ePM P

0'5 jk

ikji Tllp where ', ll incident with jee

0' jkikj

i Tlle

4 constraints (2 choices for each line ', ll )

16 synthetic constraints (4 for each jee )

Class 6 24


The 7-point tensor is defined by 11 parameters (up to scale).

A linear solution will requires 3 views (each view 4 constraints)

How many Non-linear constraints?

The number of parameters needed to describe a 7-point configurationIs 3+3=6 (because the first 5 points can be assigned the standard coordinates).

There should be 5 non-linear constraints (1 for the overallscale, and 4 additional).

Class 6 25


The dual reprojection equation:

kjkij

i pTlp 76

5

Given point 5p and a line 6l coincident with 6p

then the contraction jkij

i Tlp 65 produces the point 7p

6 image points and the shape tensor determine the 7th point

Class 6 26

Items we Skip in Class

• 8-point dual tensor: 3x3x3x3 tensor, 58 synthetic constraints, 2 views are necessary to recover tensor (first view 12 constraint, second view 11 constraints).

• Homography duality (and slices of 7-point tensor), where do the non-linear constraints come from.

Class 6 27

Duality Under a Fixed Reference Plane

Assume some plane has been identified and “stabilized” throughout thesequence of images (i.e., points on the reference plane project to fixedPoints throughout the sequence).

The house façade is stabilized The road sign is stabilized

(movie clips due to Michal Irani)

Class 6 28


In this case, the first 4 basis points are coplanar:

0

0

0

1

1P

0

0

1

0

2P

0

1

0

0

3P

0

1

1

1

4P

Class 6 29

0

0

0

1

0

0

1

M

0

0

1

0

0

1

0

M

0

1

0

0

1

0

0

M

0

1

1

1

1

1

1

M

0

0

1

1M

0

1

0

2M

1

0

0

3M

4M

00

00

00

M


1

1

1

321 MMM

(unconstrained)

Class 6 30

ii

ii

ii

i

i

i

i

i

ZW

YW

XW

W

Z

Y

X

p

00

00

00

00

00

00

....7,6,5ifor

MPMPp iiiˆˆ

M̂

)(Mnull


Class 6 31

M̂

)(Mnull


/1

/1

/1

/1

)(Mnull

(general) (ref plane stabilized)

When M̂ varies

along a linear subspace

(say a line)

)(Mnull varies

along an algebraic surface

)(Mnull varies

along a linear subspaceof the same dimension

(easy geometric interp)

Class 6 32

0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

1100

1010

1001

5̂P

ZW

YW

XW

P

00

00

00

6̂

6-point Tensor Under a Fixed Reference Plane

What is different?

056 GppT

Class 6 33

6-point Tensor Under a Fixed Reference Plane

Since 4321 ,,, PPPP are coplanar

0nPTi where Tn )1,0,0,0(

Since the bilinear constraint holds for all choices ofM

consider: TunM 0uMP P

056 GppT

0

ˆ

ˆ

ˆ

ˆ

det

6'6

66

5'5

55

Pl

Pl

Pl

Pl

T

T

T

T

'66

'55 ,,, llll coincident with u

holds for all choices of 65ˆ,ˆ PP !!

0GuuT

Gis skew-symmetric (only 3 parameters)

Class 6 34

5p

5P

6P

5Gp

056 Ge

e

C (center of proj)

065 eGT

TGG

6556 ee

There is only one (dual) epipole!

Simple Geometric Interpretation

Class 6 35

The Dual Image Ray

Recall: image ray in multi-view geometry is defined as follows:

0MPlT

0' MPl TP has a 1-parameter degree of freedom

defined by the intersection of the two planes

MlMl TT ',

Let ', ll be 2 lines coincident with MPp

this is the image ray

(the lines passing throughp and )(Mnull )

Class 6 36

The Dual Image Ray

In Dual:

0ˆˆ MPlT

0ˆˆ' MPl TM̂ has a 1-parameter degree of freedom

The line passing through p and )ˆ(Pnull

In general case: the camera center varies along a non-linear curve!

Class 6 37

The Dual Image Ray

In ref-plane stabilized: the camera center varies along a line,

and the line is the one joining p and P

ZW

YW

XW

lMPl TT

00

00

00ˆˆ0

Pnull )(Mnull

)(Mnull varies along the linepP

Class 6 38

Items we Skip in Class

• 8-point dual tensor: 3x3x3x3 tensor, 72 synthetic constraints, no non-linear constraints. 2 views are required (first contributes 5 constraints, the second 4 constraints).

• Number of Synthetic Constraints for 7-point tensor is 21. There no non-linear constraints. Only 2 views are required to solve.

Class 61 Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics...

Documents

Transcript of Class 61 Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics...