1

Combining Approximate GeometryCombining Approximate Geometrywith VDTM – A Hybrid Approach towith VDTM – A Hybrid Approach to

3D Video Teleconferencing3D Video Teleconferencing

Celso KurashimaCelso Kurashima22, Ruigang Yang, Ruigang Yang11, Anselmo , Anselmo LastraLastra11

11Department of Computer ScienceDepartment of Computer ScienceUniversity of North Carolina at Chapel HillUniversity of North Carolina at Chapel Hill

22Laboratório de Sistemas Integráveis - LSILaboratório de Sistemas Integráveis - LSIEscola Politécnica da Universidade de São PauloEscola Politécnica da Universidade de São Paulo

Fortaleza, October 8th, 2002Fortaleza, October 8th, 2002

SIBGRAPI 2002

2

IntroductionIntroduction

3

Introduction Introduction

Video ConferenceVideo Conference2D vs. 3D2D vs. 3D

1 camera1 camera Many camerasMany camerasFixed viewpontFixed viewpont Free viewpointFree viewpointNo eye contactNo eye contact Eye contactEye contactStandard VideoStandard Video Computer Vision & Computer Vision &

Computer GraphicsComputer Graphics

4

OutlineOutline

• 3D Video Teleconference System3D Video Teleconference System• Geometry ExtractionGeometry Extraction• Rendering Images with VDTMRendering Images with VDTM• ResultsResults• ConclusionConclusion

5

3D Video Teleconference 3D Video Teleconference System System (1/3)(1/3)

• Top ViewTop View

(A)(A) (B)(B)

PCCLUSTER

PCCLUSTER

6

3D-Video Teleconference System 3D-Video Teleconference System (2/3)(2/3)

Projector.

.

FirewireCameras

Geometry/Renderer PC

Screen

.

VideoCapture

PC's

LAN

7

3D-Video Teleconference System 3D-Video Teleconference System (3/3)(3/3)

• How does it work?How does it work?• 11stst: Create a geometry proxy of the : Create a geometry proxy of the

personperson• 22ndnd: Texture map images onto the proxy : Texture map images onto the proxy

with VDTM : an IBR method developed with VDTM : an IBR method developed by Debevec by Debevec et alet al..(’96, ’98)(’96, ’98)

8

Geometry Proxy Geometry Proxy Extraction Extraction (1/5)(1/5)

• The proxy : a simple geometric The proxy : a simple geometric representation of the personrepresentation of the person

• Mesh of trianglesMesh of triangles• Map textures onto the triangles facesMap textures onto the triangles faces

• Extraction: a pair of camerasExtraction: a pair of cameras• May be two of texture camerasMay be two of texture cameras

9

Geometry Proxy Extraction Geometry Proxy Extraction (2/5)(2/5)

• Algorithm- Plane + Parallax (Kumar, 94) Algorithm- Plane + Parallax (Kumar, 94) method:method:

– Robust Plane FittingRobust Plane Fitting– Stereo Feature trackingStereo Feature tracking

10

Geometry Proxy Extraction Geometry Proxy Extraction (3/5)(3/5)

• Robust Plane FittingRobust Plane Fitting– SegmentationSegmentation– Points on Points on

the Silhouettethe Silhouette– MatchingMatching– Fit PlaneFit Plane– DistanceDistance– Std. Dev.Std. Dev.– Remove distant pointsRemove distant points– RepeatRepeat

LEFT CAMERA RIGHT CAMERA

11

Geometry Proxy Geometry Proxy Extraction Extraction (4/5)(4/5)

Stereo Feature TrackingStereo Feature TrackingKLT tracker (Kanade-Lucas-Tomasi, ‘91, ‘94)KLT tracker (Kanade-Lucas-Tomasi, ‘91, ‘94)

Time

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S1: trackedfeatures

Frame t

S1: trackedfeatures

Camera 0

Camera 1

Frame t

S0: trackedfeatures

Frame t

S0: trackedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

FeatureTracking

Frame t+1

S’0: updatedfeatures

Frame t+1

S’0: updatedfeatures

Frame t+1

S’1: updatedfeatures

Frame t+1

S’1: updatedfeatures

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

EpipolarConstraint(EC)

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’0: featurespassed EC

Frame t+1

S’’1: featurespassed EC

Frame t+1

S’’1: featurespassed EC

12

Geometry Proxy Extraction Geometry Proxy Extraction (5/5)(5/5)

Triangulation & Proxy Triangulation & Proxy formation formation

+ =

13

Rendering Images with VDTM – Rendering Images with VDTM – View Dependent Texture View Dependent Texture Mapping Mapping (1/2)(1/2)

• VDTM requires a good spatial geometry VDTM requires a good spatial geometry of the objects of the objects (Debevec98)(Debevec98)

• Our system [ Our system [ Buehler Buehler et alet al (2001) and Heigl (2001) and Heigl et alet al (1999)(1999) ] ]

Spatial geometry == proxySpatial geometry == proxyTextures == live images (from Textures == live images (from

cameras)cameras)• Textures are mapped onto the proxy Textures are mapped onto the proxy

according to the viewpointaccording to the viewpoint

14

Rendering Images with VDTM – Rendering Images with VDTM – View Dependent Texture View Dependent Texture Mapping Mapping (2/2)(2/2)

• Virtual Camera at Virtual Camera at Viewpoint DViewpoint D

• Texture from Texture from cameras Ccameras Cii mapped mapped onto the triangles onto the triangles facesfaces

• Blending weights in Blending weights in vertex Vvertex V

• Angles Angles ii, used to , used to compute the weights compute the weights valuesvalues

ii = exp(- = exp(-ii22/2./2.22))

Subject's 3D proxy points

V

C0 C2 C3C1

0

1

D

2

3

15

Results Results (1/3)(1/3)

• Geometry Proxy & Image rendering with VDTMGeometry Proxy & Image rendering with VDTM

• Geometry Geometry ProxyProxy

• Image Image rendered with rendered with 2 texture 2 texture camerascameras

• Image Image rendered with rendered with 4 texture 4 texture camerascameras

16

Results Results (2/3)(2/3)

• Cameras: Firewire IEEE 1394 SONYCameras: Firewire IEEE 1394 SONY– Frame size: 320x240 pixelsFrame size: 320x240 pixels

• Geometry/Renderer PCGeometry/Renderer PC– Processor: Intel Pentium4, 2.2 GHzProcessor: Intel Pentium4, 2.2 GHz– Graphics card: nVidia GeForce3Graphics card: nVidia GeForce3

• Video Frame Rate: 3-4 fpsVideo Frame Rate: 3-4 fps

17

Results Results (3/3)(3/3)

• 3D Video Teleconferencing3D Video TeleconferencingMovieMovie

18

ConclusionsConclusions• A hybrid system for 3D Video A hybrid system for 3D Video

TeleconferencingTeleconferencing

• Fast geometry proxy extraction, using a Fast geometry proxy extraction, using a robust plane fitting method and stereo robust plane fitting method and stereo feature tracking, combined with view-feature tracking, combined with view-dependent texture mappingdependent texture mapping

• Real-time demonstration with personal Real-time demonstration with personal computers and commodity graphics cardcomputers and commodity graphics card

19

Future WorkFuture Work

• Vision-based head-tracking for Vision-based head-tracking for viewpoint controlviewpoint control

• Increase of the frame rate with faster Increase of the frame rate with faster segmentationsegmentation

20

AcknowledgementsAcknowledgements• Herman TowlesHerman Towles• Office of the FutureOffice of the Future (OOTF) group at (OOTF) group at

UNC-CHUNC-CH• U.S. Dept. of Energy and Sandia U.S. Dept. of Energy and Sandia

National Labs.National Labs.• U.S. National Science FoundationU.S. National Science Foundation• NEC/CPDIANEC/CPDIA

21

Thank you!Thank you!

http://www.cs.unc.edu/Research/ootf/http://www.cs.unc.edu/Research/ootf/http://www.lsi.usp.br/~kurash/http://www.lsi.usp.br/~kurash/