Photo VR Editor: A Panoramic and Spherical Environment Map Authoring

Tool for Image-Based VR Browsers

Jyh-Kuen Horng, Ming Ouhyoung

Communications and Multimedia Lab.

Dept. of Computer Science and Information Engineering

National Taiwan University, Taipei, Taiwan, R.O.C.

Outline Introduction

Related works

System overview - two sub-systems• Manual editor• Automatic stitching method

Conclusions & Future work

Introduction Image-based rendering becomes more and more

important

Compare with geometry-based rendering• constant rendering time regardless of scene complexity• low computational power needed• photo-realistic

How to construct a virtual environment?

Related Works Image warping

– QuickTime VR by Apple Corp.

Video clips– VideoBrush– rich frame information

Hardware sensitive– IPIX– fisheye-lens camera

System Overview

Editing Environm ent

Form C ylindrica l Im age

Apply R M & LS

Autom atic S titching

Form Spherica l Im age

3D Transform ation

Manual S titching

S titch Im ages

Load Source Photographs

Manual Editing (1/3)

Based on real 3D graphics model

• Each photograph is taken as a texture of a 3D image plane

• All kinds of affine transformation are allowed, such as translation, rotation, scaling

• Pixel color is determined by multiple hit plane

– ray casting

– bilinear interpolation

Manual Editing (2/3)

Intensity tuning• the aperture cannot be controlled

- Before intensity tuning - After intensity tuning

Manual Editing (3/3)

Form panoramic image• Gap closing : f’ = (360 - g) * f / 360

• f’ : adjusted focal length, f : original focal length g : gap angle

• easily propagate the error

c1

c2

x2

x1

blending color =xx

CxCx2

2

2

1

2

2

21

2

1 **

• Smoothing intensity discontinuity

Automatic Stitching (1/5)

Camera

p

x

• 3D point p = (X, Y, Z)• image coordinates x = (x, y, 1)

Automatic Stitching (2/5)

• 3D point p = (X, Y, Z)

• image coordinates xk, xkl

Camera

xk

p

Image k

Image l

xl

Automatic Stitching (3/5)

The relationship between p and x

• can be described using rotational model

xVRpTVRpx 11~~

The mapping between image k and l is

111~

~ lklkllkk

lk

VRVVRRVM

Mxx

……(1)

……(2)

Automatic Stitching (4/5)

We wish to minimize the squared error metric

ii

Ti

Ti

iii edJgxIxIdE

22

0''

1

~

)()()(

g : gradient, J : Jacobian matrix

After d is solved, the matrix M is adapted by

MDIM )('

D is the deformation matrix defined by d

……(3)

……(4)

Automatic Stitching (5/5)

To eliminate the ghosting

• a local search pass is introduced

• do 3-D search based on x-, y-, z-rotation

• perform an incremental update to R

0

0

0

)(

*))((

WxWy

WxWz

WyWz

X

where

RXIR

: angular velocity

……(5)

Conclusions and Future Work (1/2)

Results

Performance

Source image number Total time elapsed Time for stitching

two images

Image size

16 29.06 sec 1.816 sec 320 x 240

16 33.5 sec 2.094 sec 320 x 240

Conclusions and Future Work (2/2)

Some topics are under investigation

• An extension of the algorithm to construct spherical environment map automatically

• A faster and more robust method

• Other kind of image source (e.g. video clips)