New Image Rectification Schemes for 3D Vision Based on Sequential Virtual Rotation

Jin ZhouJune 16th, 2009

Dissertation Defense

Outline

Introduction Rectification based on Virtual Sequential Rotation Image Rectification for Stereoscopic Visualization Camera Calibration Stereoscopic View Synthesis from Monocular

Endoscopic Sequences Rapid Cones and Cylinders Modeling from Single

Images Robot Vision Conclusions and Future Work

The Geometry of 3D to 2D

Images are 2D projections of the 3D world

3D Vision – The Problem

?

How do we extract 3D information from 2D images?

? ? ??

3D of the objects

3D of the cameras

3D Vision – Applications

Augmented Reality Scene Modeling Virtual Touring 3D Imaging Robots

3D Vision – A Human Perspective

Size Linear Perspective Object Connections Stereo Motion Shading Texture

3D Vision – Computational Approaches

Different approaches use different cues Different problems requires different

approaches. Structure from Motion (SfM)

Rely on point correspondences Single View Based Modeling (SVBM)

Rely on knowledge of the scene Camera calibration

All approaches requires the images are calibarated first (either manual or automatic)

3D Vision – Practical Challenges

Camera information is unavailable Point correspondences is not reliable and

time-consuming Image resolution is limited Degeneracy

3D Vision – Limitations of Current Approaches

Distortion Degeneracy

Due to high degree of freedom of geometric models

Lack of geometric meaningMost approaches are purely based on

algebraic derivations or imaginary objects. Not accurate or not convenient (SVBM).

3D Vision – Our Contributions

Novel image rectification schemes are proposed based on sequential virtual rotation

Novel approaches are proposed for the following problems Image Rectification for Stereoscopic visualization Camera Calibration Stereoscopic View Synthesis from Monocular

Endoscopic sequences Rapid Cones and Cylinders Modeling Monocular Vision Guided Mobile Robot Navigation

3D Vision – Results of Our Approaches

No affine/projective distortion Can handle degeneracy Intuitive geometric meanings

Lead to insights of particular problems Accurate and fast

Publications Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization", Journal

of the Optical Society of America A (JOSA), November, 2008. Wenfeng Li, Jin Zhou, Baoxin Li, and M. Ibrahim Sezan, "Virtual View Specification

and Synthesis for Free Viewpoint Television", IEEE Transactions on Circuit and Systems for Video Technologies (TCSVT), (in press)

Jin Zhou, Baoxin Li, "Rapid Cones and Cylinders Modeling from a Single Calibrated Image Using Minimal 2D Control Points", Machine Vision and Applications (revision)

Jin Zhou and Baoxin Li, “Stereoscopic View Synthesis from Monocular Endoscopic Image Sequences", IEEE Transactions on Medical Imaging, (submitted)

Jin Zhou, Ananya Das, Feng Li, Baoxin Li, "Circular Generalized Cylinder Fitting for 3D Reconstruction in Endoscopic Imaging Based on MRF", In 9th IEEE Computer Society Workshop on Mathematical Methods in Biomedical Image Analysis (Joint with CVPR 2008).

Jin Zhou and Baoxin Li, "A Four Point Algorithm for Fast Metric Cone Reconstruction from a Calibrated Image", In 4th International Symposium on Visual Computing (ISVC), 2008.

Xiaolong Zhang, Jin Zhou and Baoxin Li, "Robust Two-view External Calibration by Combining Lines and Scale Invariant Point features", In 4th International Symposium on Visual Computing (ISVC) 2008.

Publications (Cont’d) Jin Zhou and Baoxin Li, “Exploiting Vertical Lines in Vision-Based Navigation for

Mobile Robot Platforms”, International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2007.

Xiaokun Li, Roger Xu, Jin Zhou, Baoxin Li, "Creating Stereoscopic (3D) Video from a 2D Monocular Video Stream", In 3rd International Symposium on Visual Computing (ISVC), 2007.

Wenfeng Li, Jin Zhou, Baoxin Li, M. Ibrahim Sezan, "Virtual View Specification and Synthesis in Free Viewpoint Television Application", 3D Data Processing, Visualization and Transmission (3DPVT), 2006.

Jin Zhou and Baoxin Li, "Image Rectification for Stereoscopic Visualization without 3D Glasses", ACM International Conference on Image and Video Retrieval (CIVR), 2006.

Jin Zhou and Baoxin Li, “Homography-based Ground Detection for a Mobile Robot Platform using a Single Camera”, International Conference on Robotics and Automation (ICRA), 2006.

Jin Zhou and Baoxin Li, “Robust Ground Plane Detection with Normalized Homography in Monocular Sequences from a Robot Platform”, International Conference on Image Processing (ICIP), 2006.

Jin Zhou and Baoxin Li, “Rectification with Intersecting Optical Axes for Stereoscopic Visualization”, International Conference on Pattern Recognition (ICPR), 2006.

What is Image Rectification?

Image rectification is a process to transform the original images to new images which have desired properties.

General Image Transformations

H?

Image transformation can be defined by a 3x3 matrix H, which is called Homography.

xx' H

Image Rectification based on Virtual Rotation

Homography of camera rotation/zooming

If we normalize the coordinates

Assume R = I

[ | ], ' ' '[ | ]P KR I C P K R I C

1 1x ' ' ' '[ | ] ' '( ) [ | ] ( ' ')( ) xP X K R I C X K R KR KR I C X K R KR

1( ' ')( )H K R KR

1x xK 1x' ' x'K

ˆ 'H R Camera orientation is determined at the same time

Advantages of the New Rectification Schemes Intuitive geometric meaning Robust

Rotation parameters can be computed by various basic image features, such as points, lines and circles.

Can be used for camera calibration. Can be used for 3D information extraction. Lead to non-distorted results

Reason: Rotation do not introduce affine/projective distortion

Rotations are Decomposed as Euler Angles

1 0 0

( ) 0 cos( ) sin( )

0 sin( ) cos( )xR

cos( ) 0 sin( )

( ) 0 1 0

sin( ) 0 cos( )yR

cos( ) sin( ) 0

( ) sin( ) cos( ) 0

0 0 1zR

Rotation Parameters Can Be Estimated by Basic Image Features

Each rotation has only one degree of freedom and thus only needs one constraint.

Example: transforming a point on to y axis

ˆ (0, , )TzR p d c

cos( ) sin( ) 0a b

tan( , )arc a b

/ 2

[ ] / 2

otherwize

[ tan( , )]arc a b Ambiguity!

Normalize

Image Rectification for Stereoscopic Visualization

The Principle of Stereoscopic (3D) Visualization

Motivation

Stereo content is scarce Stereo cameras/camcorders are expensive Common users seldom use stereo

cameras/camcorders We want to generate stereo content from

images/videos taken by common cameras

The Problem

Given two arbitrary images, rectify them so that the results look like a stereo pair.

Our Approach – Rectification based on Virtual Rotation

We can “rotate” camera to standard stereo setup.

1( )i i iH KR K R

Calibrated Case

( , , )TR r s t

1 2 1 2( ) / || ||r C C C C

( ) / || ||t r p r p

s r t

1( )i i iH KR K R

Known

UnKnown

Constraints of the stereo camera pair:1. The two cameras have the same intrinsic parameters (K)

and orientation (R)2. The camera’s optical axis is perpendicular to the baseline

(C1 – C2) i.e. the camera’s x axis has the same direction with the

baseline

K1 Any vector

Uncalibrated Case

For the uncalibrated case, all K, R and C are unknown. We can only start from the fundamental matrix and point correspondences.

Estimate H2 (homography for the second image)

2

0 / 2

0 with / 2

0 0 1 ( ) / 2

x x

y y

f p p w

K K f p p h

f w h

2R I

1( )i i iH KR K R ?

Determine R based on Sequential Virtual Rotation

2ˆ (1,0,0)TRe 12 2 2e K eConstraints:

First rotate around z axis so that the point is transformed to x axis (i.e. y = 0)

y zR R R

( , , ) ( ,0, )T TzR a b c d c

( ,0, ) ( ,0,0) (1,0,0)T T TyR d c e

[arctan( , )]z b a

[arctan( , )]y c d

Rotate around y axis so that the point is transformed to infinity.

Estimate H1

1 11 1 1 2 2 2 1 1 2( ) ( )H KR K R H K R K R H M

2 0( )TM I e v M 2[ ]F e M

1 2

2 2 0

2 0 2 0

12 0 2 2 2 2 0

12 2 1 2 0 1 2

( )

( )

( ) ( = )

T

T

T

T T T

H H M

H I e v M

H M e v M

H M H e v H H M

I H e v H M v v H

12 2 2 2 [ , , ]

(1,0,0) ( ,0,0)

T

T T

H e KRK e K r s t r

K k

1 2 2 2 0 2 0( )

0 1 0

0 0 1

TA

A

H I H e v H M H H M

a b c

H

Estimate H1

Determine a, b and c Property of standard stereo setup:

For two points with the same depth, their projection on different images should have the same distance (Points with the same depth should have the same disparity).

Approach Group points by similar disparities Then compute a, b by minimizing

2

1 1 2 2,

(|| x x || | x ' x ' ||)p

i j i jp i j A

H H H H

2

,

ˆ ˆ ˆ ˆ ˆ ˆ( ( ) ( ) ( ' '))p

i i i j i jp i j A

a x x b y y x x

0 0 0ˆ ˆ ˆ 'ax by c x

Results

Original Pair Hartley’s Method Our Method

Shear distortion

Results

Original Pair Hartley’s Method Our Method

Shrink horizontally

Camera Auto-Calibration from the Fundamental Matrix

Ti i iw K K

2 2 2 1[ ] [ ] Te w e Fw F Kruppa Equations

dual image of the absolute conic

Huang-Faugeras constraints

2 2

2 1

1( ) ( ) 0

2T T

T

trace EE trace EE

E K FK

Cons: Complex and hard to understand!Derivation for degenerate cases are purely algebraic.

Traditional Approaches:

Our Approach

12 2 2 1 1 1

1 1 1 2 2 2( , , , , , , )

T T T Sz y x y z

z y x z y

F K R R F R R R K

f f

We transform the original pair to a standard stereo pair through sequential virtual rotation and zooming

0 0 0

0 0 1

0 1 0

SF

F

7 DOFs

7 Parameters

Decomposition Illustration

Solving The Equation1

2 2 2 1 1 1

1 1 1 2 2 2( , , , , , , )

T T T Sz y x y z

z y x z y

F K R R F R R R K

f f

1 12 1 2 2 1 1

T T Sz z z y x yF R FR K R F R R K

2 1

2 1

0 0 0 0

0 1 0 0 1 0

0 0 0 0z

c a b a c

F c d c

d a b a d

1 21 2

2

2 1 1 2

1 1 2

sin( ), cos( ),cos( )

cos( ), sin( )cos( ) cos( ) cos( )

x xy

x xy y y

f da f f c

f d d db d

1 1 1 1( ) ( ,0, )Tz zR e d c

2

2 22 21 2

1 22 21 2

( )

,

x

adtg

bc

acd abdf f

bd acc cd abc

2 2 22 1 1 1

1 2 21 1

2 2 22 2 2 2

2 2 22 2

( )

( )

y

y

f c acctg

d bd acc

f c abctg

d cd abc

1 1 1 1( , , )e a b c1 1 1arctan( / )z b a

1 0Fe

Degeneracy Analysis

Degeneracy Analysis

Results of Monte Carlo Simulation

Stereoscopic View Synthesis From Monocular Endoscopic Videos

3D imaging helps to enable faster and safer surgical operations

Two view image rectification can not be applied to the new problem

Challenges: 1. Image quality is poor 2. Degeneracy

The Framework

We proved:1. Affine 3D reconstruction is

sufficient.2. Linear interpolation in

normalized disparity field is equal to linear interpolation in 3D space.

Strategy for Solving Degeneracy

We assume the initial two frames have same orientation (i.e. they are rectified)

The assumption makes the DOF of the fundamental matrix from 8 to 2!

No Assumption

Assume the two frame are rectified

Degeneracy!

Interpolation

a) Shows the disparities based on the SfM results

b) We do Delaunay triangulation and interpolate each triangle

c) We pick a set of grid points from b) and do bilinear interpolation

d) We fill holes using Laplacian interpolation and do smoothing.

Results of Synthetic Data

Ground truth

Stereo images

Final disparity image

Disparity image after triangulation

Results of Real Data

Rapid Cylinders and Cones Modeling from A Single Image

Overview of Our Approach

Goal: Rapid + Accurate Camera Calibration (Orientation Estimation)

Vertical lines Vanishing line of horizontal plane A Cone

Modeling from Image Cones (two points / four points) Cylinders (two points / four points)

The Coordinate Systems

(X,Y,Z,O) -- World Coordinate SystemY is perpendicular to the ground

(x,y,z,O) -- Camera Coordinate SystemCamera center is at the origin

Observation: Most objects are standing on the ground

Orientation Estimation from Vertical Lines

(0,1,0)T( , , )Ta b c R

First rotate around z axis so that the point is transformed to y axis (i.e. x = 0)

Rotate around x axis so that the point is transformed to infinity.

Orientation Estimation from a Cone

Rx

Rx (π/2)

RzRxRz

Edges are symmetric to y axisR Cross section is a circle

Illustration

Metric Rectification of the Ground Plane1 0 0

( / 2) 0 0 1

0 1 0g xH R

Original

Original Rectified

Rectified

Modeling Cones

0Μ ( (0, ,1) , , )Tc vc y r y

Standard view Cones on Ground General Cones

0Μ ( , ,M )R d

Cone Parameters:

2D Control Points for Cones Modeling:

Cones Modeling from the Standard View

Rectify the standard view to the ground plane view by Rx(π/2) The cross section is rectified to a circle The edge line is tangent to the circle The center and radius can be determined for any

point on the edge line

Modeling Cones Standing on the Ground

Ry

Four points Standard view

Ground view 3D Mesh

Modeling General Cones

R

Five points Standard View

3D Mesh

Modeling Cylinders

0L ( (0, ,1) , , )Tb tc y r y

0L ( , ,L )R d

Standard View Cylinders on Ground General Cylinders

Cylinder Parameters:

2D Control Points for Cones Modeling:

Modeling Cylinders Standing on the Ground

Ry

two points Standard view

Ground view 3D Mesh

Modeling General Cylinders

R

Four points Standard View

3D Mesh

Screenshots of Real Data Experiments

Exploiting Vertical Lines for Monocular Vision based Mobile Robot Navigation

In man-made environments, vertical lines are omni-present: buildings, boxes, bookshelves, cubicle walls, door frames

Many vision based systems assume the image plane is perpenticular to the ground plane

We proposed methods to rectify an image plane with general pose to be vertical based on vertical lines.

Rectified Images for Ground Plane Detection

ˆ ( / )TH R I Cn d

0 (0,1,0)Tn 0 0( ,0, )TC x z

0

0

cos( ) / sin( )ˆ 0 1 0

sin( ) / cos( )

x d

H

z d

The normalized homography of the ground plane has a special form in the rectified images:

4 DOFs

Results

Special Form

General Form

4 DOFs

8 DOFs

Ground Rectification for Mosaic based SLAM

After rectification, the relationship between any two views directly indicates their relative locations and orientations.

From vertical image plane, it’s easy to get the ground plane image

Results

Vision Based Control

Vanishing line

Occupied

OccupiedTurn angle

876

5…

910

11…

Nearest left Nearest right

Given an rectified imageFind the object to trackIdentify the obstacle or ground planeOutput a turn angle.Adjust the camera to make object always stays at the center of the image.

Movie 1Movie 2ImagesObstacles

Rectified Image

Other Potential Applications

Surveillance/Activity Recognition/ Path Planning

Different object’s size

Same object’s sizeGround is rectifiedRight angles are recoveredSpeed is reflected on the image

Conclusions and Future Works

Novel image rectification schemes are proposed in the context of exploring several practical 3D vision problems

For each of the problem, we designed novel algorithms and nice results are achieved. Moreover, we gain new insights to the problems.

In the future, we should combine different approaches and exploiting more visual cues for 3D vision problems.

Questions?

Thanks!