CalibrationCalibration

Camera CalibrationCamera Calibration

• GeometricGeometric– Intrinsics: Focal length, principal point, distortionIntrinsics: Focal length, principal point, distortion

– Extrinsics: Position, orientationExtrinsics: Position, orientation

• RadiometricRadiometric– Mapping between pixel value and scene radianceMapping between pixel value and scene radiance

– Can be nonlinear at a pixel (gamma, etc.)Can be nonlinear at a pixel (gamma, etc.)

– Can vary between pixels (vignetting, cosCan vary between pixels (vignetting, cos44, etc.), etc.)

– Dynamic range (calibrate shutter speed, etc.)Dynamic range (calibrate shutter speed, etc.)

Geometric Calibration IssuesGeometric Calibration Issues

• Camera ModelCamera Model– Orthogonal axes?Orthogonal axes?

– Square pixels?Square pixels?

– Distortion?Distortion?

• Calibration TargetCalibration Target– Known 3D points, noncoplanarKnown 3D points, noncoplanar

– Known 3D points, coplanarKnown 3D points, coplanar

– Unknown 3D points (structure from motion)Unknown 3D points (structure from motion)

– Other features (e.g., known straight lines)Other features (e.g., known straight lines)

Geometric Calibration IssuesGeometric Calibration Issues

• Optimization methodOptimization method– Depends on camera model, available dataDepends on camera model, available data

– Linear vs. nonlinear modelLinear vs. nonlinear model

– Closed form vs. iterativeClosed form vs. iterative

– Intrinsics only vs. extrinsics only vs. bothIntrinsics only vs. extrinsics only vs. both

– Need initial guess?Need initial guess?

Caveat - 2D Coordinate SystemsCaveat - 2D Coordinate Systems

• yy axis up axis up vs. vs. yy axis down axis down

• Origin at center vs. Origin at center vs. cornercorner

• Will often write (Will often write (uu,, vv) for image ) for image coordinatescoordinates

uu

vv

uu

uu

vvvv

Camera Calibration – Example 1Camera Calibration – Example 1

• Given:Given:– 3D 3D 2D correspondences 2D correspondences

– General perspective camera model (no distortion)General perspective camera model (no distortion)

• Don’t care about “Don’t care about “zz” after transformation” after transformation

• Homogeneous scale ambiguity Homogeneous scale ambiguity 11 free 11 free parametersparameters

lkzjyix

hgzfyexlkzjyix

dczbyax

z

y

x

lkji

hgfe

dcbashomogeneou

divide

1

lkzjyix

hgzfyexlkzjyix

dczbyax

z

y

x

lkji

hgfe

dcbashomogeneou

divide

1

Camera Calibration – Example 1Camera Calibration – Example 1

• Write equations:Write equations:

1

111

111

1111

111

vlkzjyix

hgzfyex

ulkzjyix

dczbyax

1

111

111

1111

111

vlkzjyix

hgzfyex

ulkzjyix

dczbyax

Camera Calibration – Example 1Camera Calibration – Example 1

• Linear equationLinear equation

• Overconstrained (more equations than unknowns)Overconstrained (more equations than unknowns)

• Underconstrained (rank deficient matrix – any Underconstrained (rank deficient matrix – any multiple of a solution, including 0, is also a multiple of a solution, including 0, is also a solution)solution)

0

l

c

b

a

uzuyuxuzyx

uzuyuxuzyx

uzuyuxvzyx

uzuyuxuzyx

2222222222

2222222222

1111111111

1111111111

10000

00001

10000

00001

0

l

c

b

a

uzuyuxuzyx

uzuyuxuzyx

uzuyuxvzyx

uzuyuxuzyx

2222222222

2222222222

1111111111

1111111111

10000

00001

10000

00001

Camera Calibration – Example 1Camera Calibration – Example 1

• Standard linear least squares methods Standard linear least squares methods forforAx=0 will give the solution x=0Ax=0 will give the solution x=0

• Instead, look for a solution with |x|= 1Instead, look for a solution with |x|= 1

• That is, minimize |Ax|That is, minimize |Ax|22 subject to |x| subject to |x|22=1=1

Camera Calibration – Example 1Camera Calibration – Example 1

• Minimize |Ax|Minimize |Ax|22 subject to |x| subject to |x|22=1=1

• |Ax||Ax|22 = (Ax) = (Ax)TT(Ax) = (x(Ax) = (xTTAATT)(Ax) = x)(Ax) = xTT(A(ATTA)xA)x

• Expand x in terms of eigenvectors of Expand x in terms of eigenvectors of AATTA:A:

x = x = 11ee11+ + 22ee22+…+…

x xTT(A(ATTA)x = A)x = 111122++2222

22+…+…

|x||x|22 = = 1122++22

22+…+…

Camera Calibration – Example 1Camera Calibration – Example 1

• To minimizeTo minimize1111

22++222222+…+…

subject tosubject to 11

22++2222+… = 1+… = 1

set set minmin= 1 and all other = 1 and all other ii=0=0

• Thus, least squares solution is Thus, least squares solution is eigenvector corresponding to minimum eigenvector corresponding to minimum (non-zero) eigenvalue of A(non-zero) eigenvalue of ATTAA

Camera Calibration – Example 2Camera Calibration – Example 2

• Incorporating additional constraints intoIncorporating additional constraints intocamera modelcamera model– No shear (No shear (uu, , vv axes orthogonal) axes orthogonal)

– Square pixelsSquare pixels

– etc.etc.

• Doing minimization in image spaceDoing minimization in image space

• All of these impose All of these impose nonlinearnonlinear constraints constraints ononcamera parameterscamera parameters

Camera Calibration – Example 2Camera Calibration – Example 2

• Option 1: nonlinear least squaresOption 1: nonlinear least squares– Usually “gradient descent” techniquesUsually “gradient descent” techniques

– e.g. Levenberg-Marquardte.g. Levenberg-Marquardt

• Option 2: solve for general perspective Option 2: solve for general perspective model, find closest solution that model, find closest solution that satisfies constraintssatisfies constraints– Use closed-form solution as initial guess forUse closed-form solution as initial guess for

iterative minimizationiterative minimization

Radial DistortionRadial Distortion

• Radial distortion can not be representedRadial distortion can not be representedby matrixby matrix

• ((ccuu, , ccvv) is image center,) is image center,

uu**imgimg= = uuimgimg– – ccuu, , vv**

imgimg= = vvimgimg– – ccvv,,

is first-order radial distortion coefficient is first-order radial distortion coefficient

)(1

)(12*2**

2*2**

imgimgimgvimg

imgimgimguimg

vuvcv

vuucu

)(1

)(12*2**

2*2**

imgimgimgvimg

imgimgimguimg

vuvcv

vuucu

Camera Calibration – Example 3Camera Calibration – Example 3

• Incorporating radial distortionIncorporating radial distortion

• Option 1:Option 1:– Find distortion first (e.g., straight lines inFind distortion first (e.g., straight lines in

calibration target)calibration target)

– Warp image to eliminate distortionWarp image to eliminate distortion

– Run (simpler) perspective calibrationRun (simpler) perspective calibration

• Option 2: nonlinear least squaresOption 2: nonlinear least squares

Calibration TargetsCalibration Targets

• Full 3D (nonplanar)Full 3D (nonplanar)– Can calibrate with one imageCan calibrate with one image

– Difficult to constructDifficult to construct

• 2D (planar)2D (planar)– Can be made more accuracteCan be made more accuracte

– Need multiple viewsNeed multiple views

– Better constrained than full SFM problemBetter constrained than full SFM problem

Calibration TargetsCalibration Targets

• Identification of featuresIdentification of features– ManualManual

– Regular array, manually seededRegular array, manually seeded

– Regular array, automatically seededRegular array, automatically seeded

– Color coding, patterns, etc.Color coding, patterns, etc.

• Subpixel estimation of locationsSubpixel estimation of locations– Circle centersCircle centers

– Checkerboard cornersCheckerboard corners

Calibration Target w. CirclesCalibration Target w. Circles

3D Target w. Circles3D Target w. Circles

Planar Checkerboard TargetPlanar Checkerboard Target

[Bouguet][Bouguet]

Coded CirclesCoded Circles

[Marschner et al.][Marschner et al.]

Concentric Coded CirclesConcentric Coded Circles

[Gortler et al.][Gortler et al.]

Color Coded CirclesColor Coded Circles

[Culbertson][Culbertson]

Calibrating ProjectorCalibrating Projector

• Calibrate cameraCalibrate camera

• Project pattern onto a known objectProject pattern onto a known object(usually plane)(usually plane)– Can use time-coded structured lightCan use time-coded structured light

• Form (Form (uuprojproj, , vvprojproj, , xx, , yy, , zz) tuples) tuples

• Use regular camera calibration codeUse regular camera calibration code

• Typically lots of keystoning relative to Typically lots of keystoning relative to camerascameras

Multi-Camera GeometryMulti-Camera Geometry

• Epipolar geometry – relationship Epipolar geometry – relationship between observed positions of points in between observed positions of points in multiple camerasmultiple cameras

• Assume:Assume:– 2 cameras2 cameras

– Known intrinsics and extrinsicsKnown intrinsics and extrinsics

Epipolar GeometryEpipolar Geometry

PP

CC11 CC22

pp22pp11

Epipolar GeometryEpipolar Geometry

PP

CC11 CC22

pp22pp11

ll22

Epipolar GeometryEpipolar Geometry

PP

CC11 CC22

pp22pp11

ll22

Epipolar lineEpipolar line

EpipolesEpipoles

Epipolar GeometryEpipolar Geometry

• Goal: derive equation for lGoal: derive equation for l22

• Observation: P, CObservation: P, C11, C, C22 determine a plane determine a plane

PP

CC11 CC22

pp22pp11

ll22

Epipolar GeometryEpipolar Geometry

• Work in coordinate frame of CWork in coordinate frame of C11

• Normal of plane is T Normal of plane is T Rp Rp22, where T is , where T is

relative translation, R is relative rotationrelative translation, R is relative rotation

PP

CC11 CC22

pp22pp11

ll22

Epipolar GeometryEpipolar Geometry

• pp11 is perpendicular to this normal: is perpendicular to this normal:

p p11 (T (T Rp Rp22) = 0) = 0

PP

CC11 CC22

pp22pp11

ll22

Epipolar GeometryEpipolar Geometry

• Write cross product as matrix Write cross product as matrix multiplicationmultiplication

PP

CC11 CC22

pp22pp11

ll22

0

0

0

, **

xy

xz

yz

TT

TT

TT

xxT TT

0

0

0

, **

xy

xz

yz

TT

TT

TT

xxT TT

Epipolar GeometryEpipolar Geometry

• pp11 T T* * R pR p22 = 0 = 0 p p11TT EE pp22 = 0 = 0

• E is the E is the essential matrixessential matrix

PP

CC11 CC22

pp22pp11

ll22

Essential MatrixEssential Matrix

• E depends only on camera geometryE depends only on camera geometry

• Given E, can derive equation for line lGiven E, can derive equation for line l22

PP

CC11 CC22

pp22pp11

ll22

Fundamental MatrixFundamental Matrix

• Can define Can define fundamental matrixfundamental matrix F F analogously, operating on pixel analogously, operating on pixel coordinates instead of camera coordinates instead of camera coordinatescoordinates

uu11TT FF uu22 = 0 = 0

• Advantage: can sometimes estimate F Advantage: can sometimes estimate F without knowing camera calibrationwithout knowing camera calibration