Visualization of Urban Scenes from Video Footages

Siyuan Fang

Supervised by Dr. Neill Campbell 29/10/2007

Overview

Generate panorama (or image mosaicing) from a video of street scenes

Potential applications: Virtual environment Robot navigation On-line tourist guide

Combine different source frames (or images) together

Outline

• Background • Research Proposal • Achievements • Future Plan

Mosaicing form Single-Viewpoint Images

Source images are captured using a camera with pure rotation movement around a fixed viewpoint, (the intrinsic parameter can also change)

Align (or register) source images and stitch them together, e.g., the general alignment (registration) model: homography for a pair of images.

The homography can be estimated by points (pixels or features) correspondence, robust estimations (e.g., RANSAC) are used to remove the outliers of the point correspondence

Further refinements: pixel value blending bundle adjustment

The limitation of the single-viewpoint image mosaicing--cannot put viewpoint far away enough

Mosaicing form Multi-Viewpoint Images : Perspective Mosaicing

• The difficulty: there isn’t a single uniform alignment --only planar scenes have the same homography if viewpoint is changing

• Traditional approaches: view synthesis and image based rendering e.g., [GORT96], [SHUM99]--picture surface alignment + residual pixel movement [KUMA95]

• User-interactive design: picture surface alignment + user refinement for the off-plane scenes [AGAR06]

• Limitations of these approaches

Strip Panorama: Parallel Projection Mosaicing

A simulation of the push-broom camera. A common setup: sideway translating camcorder and vertical trip scanning. [ZHEN03]

Parallel projection along the horizontal direction, hence, remain the perspective along the vertical direction, cause the aspect ratio distortion:

Further scenes get expanded Closer scenes get squashed

Strip Panorama: Parallel Projection Mosaicing Cont.

Two kinds of distortion, the same reason:

Aspect ratio distortion Blurred-truncated

distortion

A Better Strip Panorama: Crossed-Slit Mosaicing

The Crossed-Slit Cameral Model, and its special case: POX-slit model.

Strict sideway translation camcorder can be used to simulate POX-slit model: the cameral path is presumably considered as the horizontal slit and the vertical slit is determined by the camera path and the width of the image plane [ZOME03]

The Crossed-Slit Mosaicing cont.

Comparison between the Crossed-Slit Mosaicing and the Parallel projection Mosaicing

The Limitations of the Crossed-Slit Mosaicing (Distortions)

Aims and Objectives

Aims and basic requirements: . Use moving pinhole camcorder (normal) Capture real long street scenes Scalable Least distortion Automaticity

The Proposed Research Approach (provisional)

The proposed approach, 3 main tasks:I. Synthesis novel view based on the crossed-slit mosaic, but fully perspective

— remove two kinds of distortion based on geometry information

II. Handle extremely long scenes,

III. * Handle moving objects

Summary of Achievements

The literature survey on image mosaicing Single-viewpoint mosaicing View synthesis and imaged based rendering Approaches of strip mosaicng

A (provisional) research proposal

Some design and implementation work A single-viewpoint mosaic tool A part of the task 1

Future Plan

Further refine (modify) the research proposal

Complete the task 1 A formal math model and an

approximation algorithm Curved camera path and corner

effect Writing up

Start the design work of task 2 The main problem User interactive may be needed The result of task 1 can be

served as the building block

Reference

[AGAR06] A.Agarwala, M.Agrawala, M.Cohen and D.Salesin, R.Szeliski, Photographing long scenes with multi-viewpoint panoramas, ACM Transactions on Graphics, 25(3), 853 - 861, 2006.

[GORT96] S.J.Gortler, R.Grzeszczuk, R. Szeliski and M.F. Cohen, The Lumigraph, The Journal of Computer Graphics, 30, 43--54, 1996.

[KUMA95] R.Kumar, P.Anandan, M.Irani, J.Bergen and K.Hanna, Representation of scenes from collections of images, Proceedings of IEEE Workshop on Representation of Visual Scenes, 10-17 ,1995

[PELE00] S.Peleg, B.Rousso, A.Rav-Acha and A.Zomet, Mosaicing on Adaptive Manifold, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10), 1144-1154, 2000.

[SHUM99] H.Y.Shum and L.W.He, Rendering with concentric mosaics, The Journal of Computer Graphics, 33, 299--306, 1999.

[ZHEN03] J.Y.Zheng, Digital route panoramas, IEEE Multimedia, 10(3), 57- 67, 2003.

[ZOME03] A.Zomet, D.Feldman, S.Peleg and D.Weinshall, Mosaicing New Views: The Crossed-Slits Projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 741- 754, 2003.