CS 395/495-25: Spring 2003 IBMR: Week 7A The Camera Matrix Continued... Jack Tumblin...

CS 395/495-25: Spring 2003CS 395/495-25: Spring 2003

IBMR: Week 7A IBMR: Week 7A

The Camera MatrixThe Camera MatrixContinued...Continued...

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

IBMR-Related SeminarsIBMR-Related Seminars

3D Scanning for Cultural Heritage Applications3D Scanning for Cultural Heritage Applications Holly Rushmeier, IBM TJ Watson Holly Rushmeier, IBM TJ Watson

FridayFriday May 16 3:00pm, Rm 381, CS Dept. May 16 3:00pm, Rm 381, CS Dept.

no title yet ...<BRDF, BRSSDF capture? no title yet ...<BRDF, BRSSDF capture? Optics of Hair? Inverse Rendering?>Optics of Hair? Inverse Rendering?>

Steve Marschner, Cornell University Steve Marschner, Cornell University Friday May 23 3:00pm, Rm 381, CS Dept.Friday May 23 3:00pm, Rm 381, CS Dept.

RemindersReminders

• ProjA graded: Good Job! 90,95, 110ProjA graded: Good Job! 90,95, 110• ProjB graded: Good! minor H confusions...ProjB graded: Good! minor H confusions...• MidTerm graded:MidTerm graded:• ProjC posted tonight; Due next Friday, May 15ProjC posted tonight; Due next Friday, May 15

• Start Watson’s Late Policy: Grade -(3Start Watson’s Late Policy: Grade -(3nn) points;) points;

n=# of class meetings late n=# of class meetings late• ProjD coming Thurs May 15, due Thurs May 29ProjD coming Thurs May 15, due Thurs May 29• Take-Home Final Exam: Thurs June 5, due June 11Take-Home Final Exam: Thurs June 5, due June 11

Basic Cameras RevisitedBasic Cameras Revisited

ffCC

yycc

zzcc

xxcc

pp

Plenty of TerminologyPlenty of Terminology::

• Image Plane or Focal PlaneImage Plane or Focal Plane

• Focal Distance Focal Distance ff

• Camera Center Camera Center CC

• Principal Point Principal Point pp

• Principal Axis Principal Axis zzcc

• Principal Plane Principal Plane (?!?! DOESN’T touch principle point!?!?)(?!?! DOESN’T touch principle point!?!?)

• Camera Coords (xCamera Coords (xcc,y,ycc,z,zcc))

• Image Coords (x’,y’)Image Coords (x’,y’)

Recall Basic Camera PRecall Basic Camera P00: :

• Basic Camera as a 3x4 matrix:Basic Camera as a 3x4 matrix:

• To translate To translate imageimage origin ( origin (x’,y’x’,y’) away from ) away from zzcc axis: axis:

Shift principal point Shift principal point pp from ( from (0,00,0) to () to (ppxx,p,pyy))

– does does NOTNOT use ‘homogeneous’ use ‘homogeneous’ 11 term in term in X X

– is is NOTNOT obvious: scales obvious: scales zzc c (think in P(think in P22 z zcc=x3 = 1,...)=x3 = 1,...)

x’x’y’y’z’z’

xxcc

yycc

zzcc

11

==

PP00 X = x X = x

yyy’y’

ffzz

CC

yycc

xxcc

zzcc

pp

ff 0 0 ppxx 00

0 0 ff ppyy 00

0 0 0 0 11 0 0

Basic Camera PBasic Camera P00: P: P33PP22 ((or camera or camera RR33))

• Basic Camera PBasic Camera P0 0 is a 3x4 matrix:is a 3x4 matrix:

• Non-square pixels? change scaling Non-square pixels? change scaling ((xx, , yy))

• Parallelogram pixels? set nonzero skew Parallelogram pixels? set nonzero skew ss

K matrixK matrix: “(internal) camera calib. matrix”: “(internal) camera calib. matrix”

xxyyzz

xxcc

yycc

zzcc

11

==

PP00 X = x X = x

yyy’y’

ffzz

CC

yycc

xxcc

zzcc

pp

xxff ss ppxx 00

0 0 yyff ppyy 00

0 0 0 0 11 0 0

KK (3x3 submatrix)(3x3 submatrix)

[K 0] = P[K 0] = P00

Complete Camera Matrix PComplete Camera Matrix P

• K matrixK matrix: “: “internalinternal camera calib. matrix” camera calib. matrix”

• RR··T matrixT matrix: “: “externalexternal camera calib. matrix” camera calib. matrix” – T matrix: Translate world origin to cam. origin,T matrix: Translate world origin to cam. origin,– R matrix: then Rotate world to fit cam. axesR matrix: then Rotate world to fit cam. axes

Combine: writeCombine: write

(P(P00··RR··T)T)··X = X = xx

asas

PP··X = X = xx

Output: Output: xxPP22 Camera Image Camera Image

Input:Input: XXPP33 World Space World Space

zz

yy

xx

XX (world space)(world space)

zz cc

yy cc

xx cc

CC

xx(c

amer

a sp

ace)

(cam

era

spac

e)

The Pieces of Camera Matrix The Pieces of Camera Matrix PP

ColumnsColumns of P matrix = image of world-space axes: of P matrix = image of world-space axes:

• pp11,p,p22,p,p33 == image of x,y,z axis vanishing points == image of x,y,z axis vanishing points

– Direction D = [1 0 0 0]Direction D = [1 0 0 0]TT = point on P = point on P33’s x’s x11 axis, at inifinity axis, at inifinity

– PD = 1PD = 1stst column of P = P column of P = P11. Repeat for y and z axes. . Repeat for y and z axes.

• pp44 == image of the world-space origin pt. == image of the world-space origin pt.– Proof: let Proof: let DD = [0 0 0 1]T = world origin = [0 0 0 1]T = world origin– PDPD = 4 = 4thth column of column of PP = image of origin pt. = image of origin pt.

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

pp11 pp22 pp33 pp44

ff CC

yycc

zzcc

xxcc

pp


RowsRows of P matrix: camera planes in world space of P matrix: camera planes in world space• row 1 = row 1 = PP1T1T = image x-axis plane = image x-axis plane• row 2 = row 2 = PP2T2T = image y-axis plane = image y-axis plane• row 1 = row 1 = PP3T3T = camera’s principal plane = camera’s principal plane

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

PP1T1T

PP2T2T

PP3T3T

ff CC

yycc

zzcc

xxcc

pp


RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• row 1 = row 1 = PP33 = camera’s principal plane = camera’s principal plane

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

ff CC

yycc

zzcc

xxcc

pp

PP1T1T

PP2T2T

PP3T3T


RowsRows of P matrix: planes in world space of P matrix: planes in world space• row 1 = row 1 = PP11 = image x-axis plane = image x-axis plane• row 2 = row 2 = PP22 = image y-axis plane = image y-axis plane• Careful!Careful! Shifting the Shifting the

image origin by image origin by ppxx, p, pyy

shifts the x,y axis planes!shifts the x,y axis planes!

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

ff CC

yycc

zzcc

xxcc

pp

PP1T1T

PP2T2T

PP3T3T



xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

ff CC

yycc

zzcc

xxcc

pp

PP1T1T

PP2T2T

PP3T3T



– princip. plane princip. plane PP33= = [p[p3131 p p3232 p p3333 p p3434]]TT

– its normal direction: its normal direction: [p[p3131 p p3232 p p33 33 0 ]0 ]TT

– Why is it normal? It’s the world-space Why is it normal? It’s the world-space P P33 direction of the direction of the zzcc axis axis

BOOK has yet another inconsistent notation for this.....BOOK has yet another inconsistent notation for this.....

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

ff CC

yycc

zzcc

xxcc

pp

Principal Principal plane pplane p33

PP1T1T

PP2T2T

PP3T3T


• Principal Axis Vector (zPrincipal Axis Vector (zcc)) in in worldworld space: space:– Normal of principal plane: = Normal of principal plane: = mm33 = = [p[p3131 p p3232 p p33 33 ]]TT

– PP33 Scaling Scaling Ambiguous direction!! +/- m Ambiguous direction!! +/- m3 3 ? ? – Solution: use Solution: use det(M)det(M)··mm33 as front of cameraas front of camera

• Principal Point Principal Point p p in in imageimage space: space:– image of (infinity point on zimage of (infinity point on zcc axis = m axis = m33))

MM··mm33 = = pp = x = x0 0 (book renames p as x(book renames p as x00))

CC

yycc

zzcc

xxcc

pp

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

pp44

MM

mm11TT

mm22TT

mm33TT

xx00 ff

••••••00

••••••11


Where is the Camera?Where is the Camera?

• Camera center Camera center CC is at camera origin is at camera origin(x(xcc,y,ycc,z,zcc)=0=C)=0=C

• What point What point CC will the camera will the camera PP transform to transform to C=0 C=0 ??Answer: Answer: CC is the Null Space of is the Null Space of P :P :

P C = C = 0 P C = C = 0 (solve for C. SVD works, but here’s an easier way:)(solve for C. SVD works, but here’s an easier way:)

Finite Camera: Finite Camera: C = C = Affine Camera: C Affine Camera: C = =

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or

~~~~ --MM-1-1··pp44 dd(where (where Md=0)Md=0)

~~

~~

PP = =• • •• • • ••• • •• • • ••• • • • • • • •

MM

mm11TT

mm22TT

mm33TT

ff CC

yycc

zzcc

xxcc

pp

pp44

Uses for Camera Matrix Uses for Camera Matrix PP

Each world space Each world space RR33 point point d = (x,y,z)d = (x,y,z) defines a world-space defines a world-space PP33 directiondirection D = (x,y,z,0)D = (x,y,z,0)..

What is the image point What is the image point xxdd from direction from direction DD? ?

xxdd = PD = [M | p= PD = [M | p44] D = M] D = M dd

((PP44 column has no effect, because of D’s zero): column has no effect, because of D’s zero):

xxdd = M = M d or Md or M-1-1xxdd = d = dff CC

yycc

zzcc

xxcc

pp

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

MM

xxdd

~~

pp44

zzyy xx

XX (world space)(world space)

dd


Given image pointGiven image point xx00 and camera matrix and camera matrix PP,,Find ray Find ray X(X()) in world space through both: in world space through both:Slow, Obvious way: Slow, Obvious way:

‘‘Pseudo-invert’ Pseudo-invert’ PP, apply to x, apply to x00::• Define pseudo-inverse Define pseudo-inverse PP++ as as = P= PTT(PP(PPTT))-1-1

(note (note P·PP·P++ = I = I))• Find a world-space point on ray: XFind a world-space point on ray: X0 0 = P= P+ + xx00

• LIRP with world-space camera point C: LIRP with world-space camera point C:

X(X() = C + (X) = C + (X00-C)-C)ff CC

yycc

zzcc

xxcc

pp

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

MM

xx00

X(X())

~~

pp44

~~ ~~~~

~~


Given image pointGiven image point xx00 and camera matrix and camera matrix PP,,Find ray Find ray X(X()) in world space through both: in world space through both:

Better way: ray from Better way: ray from CC to point to point DD00 at infinity: at infinity:• Point Point xx00 is the image of is the image of (unknown)(unknown) world-space direction world-space direction

DD00 = (x,y,z,0) = (x,y,z,0)TT. Define a point . Define a point dd00 = (x,y,z) = (x,y,z)TT. . • Recall: we can find Recall: we can find dd00 from the image: from the image: MM-1-1xx00 = d = d00 • Recall: world-space camera Recall: world-space camera C = -MC = -M-1-1pp44

X(X() = ) = MM-1-1xx00 + C = M + C = M-1-1 ((xx00 – – pp44))

0 1 10 1 1

ff CC

yycc

zzcc

xxcc

pp

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

MM

xx00

X(X())

~~

pp44

~~

~~

~~


Given world-space pointGiven world-space point XX00, camera matrix , camera matrix PP,,

Find camera depth Find camera depth zz00::

– XX00 = [x = [xww, y, yww, z, zww, t, tww]]TT seen thru camera seen thru camera PP is is

XX00··PP = = xx00 = [x = [xcc, y, ycc, 1], 1]TT··wwcc

– Then Then signedsigned depth depth zz00 is: is:

zz00 = = wwcc sign(det sign(det MM))

ttww || || mm33 || ||

ff CC

yycc

zzcc

xxcc

pp

xxcc

yycc

zzcc

xxww

yyww

zzww

ttww

==• • • •• • • •• • • •• • • •• • • •• • • •

PPPP··X = xX = x, or, or PP = =• • •• • • ••• • •• • • ••• • • • • • • •

xx00

XX00

zz00

pp44

MM

mm11TT

mm22TT

mm33TT

Skipped:Skipped:

• P = [K|0]P = [K|0]··RR··TT How can we separate K, R, T? How can we separate K, R, T?– Answer: K is triangular; use QR decompositionAnswer: K is triangular; use QR decomposition

• Cameras at Infinity:Cameras at Infinity:– Orthographic or ‘Parallel Projection’ CamerasOrthographic or ‘Parallel Projection’ Cameras– Transition to Orthographic:Transition to Orthographic:

• Weak Perspective projection camerasWeak Perspective projection cameras• the ‘zoom’ lens (variable f)the ‘zoom’ lens (variable f)

– Moving line-scan or ‘pushbroom’ cameras Moving line-scan or ‘pushbroom’ cameras • Translation Scan: aerial/sattelite camerasTranslation Scan: aerial/sattelite cameras• Cylindrical Scan: panoramic camerasCylindrical Scan: panoramic cameras• UNC ‘HiBall Tracker’: 6 tiny self-locating line-scan camerasUNC ‘HiBall Tracker’: 6 tiny self-locating line-scan cameras

Chapter 6 Chapter 6 In Just One Slide:In Just One Slide:

Given point correspondence sets Given point correspondence sets (x(xii X Xii)), How , How do you find camera matrix do you find camera matrix P ? P ? (full 11 DOF)(full 11 DOF)

Surprise! You already know how !Surprise! You already know how !• DLT method: DLT method:

-rewrite H x = x’ as Hx -rewrite H x = x’ as Hx x’ = 0 x’ = 0--rewrite rewrite P X = xP X = x as as PX PX x = 0 x = 0-vectorize, stack, solve Ah = 0 for h vector-vectorize, stack, solve Ah = 0 for h vector-vectorize, stack, solve -vectorize, stack, solve Ap = 0Ap = 0 for for pp vector vector

--Normalizing step removes origin dependenceNormalizing step removes origin dependence

• More data More data better results (at least 28 point pairs) better results (at least 28 point pairs)• Algebraic & Geometric Error, Sampson Error…Algebraic & Geometric Error, Sampson Error…

ENDEND

CS 395/495-25: Spring 2003 IBMR: Week 7A The Camera Matrix Continued... Jack Tumblin...

Documents

Transcript of CS 395/495-25: Spring 2003 IBMR: Week 7A The Camera Matrix Continued... Jack Tumblin...