Real-time View Synthesis for Virtual Reality

and Light Field displays

Prof. Gauthier Lafruit

Sarah Fachada

Daniele Bonatto

Yan Li

Arnaud Schenkel

Henry-Louis Guillaume

Ségolène Rogge

Prof. Adrian Munteanu

2

Acquisition, Synthesis & Rendering in XR

XR, light field displays & hologramsCamera arrays, plenopic & GLC cameras

DIBR:Depth Image-

Based Rendering

DERSRVS

3DoF+6DoF

MPEG-I immersive media formats3D Graphics 3D meshes + 2D textures

Point CloudsOctree + modified Video Codec

Light FieldsMultiview + Depth, DIBR

2020

2020-2022

MPEG-I (Immersive)

5

Lafruit et. al

© ULB

Brussels 3DLicorneA project

Acquisition: moving camera (X,Y,Z translations)

6

Acquisitions along green and blue planes= ULB_Unicorn (in MPEG-I)

Video Acquisition& View Synthesis

15 x 4k Black Magic Cameras

How ??

7

Camera 0 viewLeft re-projection Right re-projection

Camera 0 Left re-projectionHole filling from

Camera 1 re-projection

Blended

Camera View Virtual View

Point Cloud

Synthesis

Depth map

Triangles between pixel triplets to help hole filling

8

RVS 3.1: Reference View Synthesizer

N Input views with associated depth maps9

Challenge: Depth estimation

DERS: Depth Estimation Reference Software

Bad Depth map

10

Virtual Reality with 4-views RVS

12Lafruit et. al© ULB

Virtual Reality: 4-views RVS on ULB_Unicorn

13

Light Field display: 72 output views, Holografika @VUB

14

72 viewpoints for horizontal parallax15

Light Field display: 72 output views, Holografika @VUB

Hologram 4-views RVS

17

Hologram Photogrammetry: 500 input pictures!

Fur objects (goat, sheep) at the left have been removed from the hologram, because of bad quality in photogrammetry

18

RVS

Holo-grams

Ref

4-views RVS 8-views RVSComparisons

20

4-views RVSPhotogrammetry (500 pic.)C

om

par

iso

ns

21

RVS: 360 Dataset Museum, perfect depth map

22

RVS: Dataset Fencing, moderate depth quality

23

Conclusions

• View Synthesis RVS is mature for 3DoF+/6DoF XR

• Real-time RVS on GPU

• Depth Estimation DERS still requires some attention

• How to define quality for Light Field displays and Holography? (no ref for PSNR, SSIM, etc)

24

Reference softwareEduardo Juarez et. al., Manual of Depth Estimation Reference Software (DERS 8.0), ISO/IEC JTC 1/SC 29/WG 11 N18450, Geneva, Switzerland, March 2019.

http://mpegx.int-evry.fr/software/MPEG/Explorations/6DoF/DERS.git

Bart Kroon et. al, Reference View Synthesizer (RVS) manual, ISO/IEC JTC1/SC29/WG11 MPEG/N18068, Macau SAR, China, October 2018.

http://mpegx.int-evry.fr/software/MPEG/Explorations/3DoFplus/RVS/tree/v3.1

25

Video Point Cloud Coding (V-PCC)

Lossy coding = 20 Mbps Lossless w/o attributes (geometry only) = 40 – 66 Mbps (1.5 bpv)Lossless (all attributes) = 200 – 400 Mbps (10 bpv)Geometry = 15 - 20% , Color = 70 - 84% , Patches = 1 - 10% of total bitrate

Orthographic projectionBounding volume

Color Patchesclustered in a video

Depth Patchesclustered in a video

Segmentation in Patches

26

AnimatedPoint Cloud

(voxels)

Point Cloud Coding: Quality vs. Bitrate

2.5 Mbps 6 Mbps 19.5 Mbps 3.5 Mbps 8 Mbps 26.5 Mbps

Lafruit et. al27

Augmented Reality with Point Clouds

https://www.3ders.org/articles/20180305-russians-take-ar-selfies-with-40-ft-vladimir-putin.html

http://research.microsoft.com/holoportation

28Lafruit et. al

6DoF with possibility to turn around the objects/persons