1

Anahita: A System for 3D Video Streaming with Depth Customization

K. Calagari, K. Templin, T. Elgamal, K. Diab, P. Didyk, W. Matusik, M. Hefeeda

QCR & MIT & SFU5 November 2014

2

3D content is popular especially in theaters

Motivations

Controlled, homogenous environments

3

… is a lot more challenging because of …

complexity of perceiving and rendering depth

on different displays

&

in dynamic network conditions

Internet 3D Video Streaming …

Sample 3D Displays

4

How do we see 3D?

5

6

Approximate Model for Depth [Holliman 04]

d

e

Left eye

Right eye

p

Left eye

Positive disparity

zd

e

Right eye

p

z

Negative disparity

Comfort zone depends on:

• Presented content• Viewing condition• Screen distance• …

“The zone of comfort: Predicting visual discomfort with stereo displays” by Shibata et al. 2011

Depth & Comfort Zone [Shibata 11]

7

8

Many different display technologies - From anaglyph to auto stereoscopic

Rendering 3D Content

3D Display: Active/Passive Glasses

9

Autostereoscopic

• Parallax Barriers:

• Blocks light in certain direction

3D Display: Glasses-Free

• Lenticular Lens:

• Reflects light to certain direction

10

3D Display: Multi-views

11

12

Dynamic conditions- Bandwidth- Loss rate- Delay

Network Dynamics

13

Depth perception heavily depends on - Display size and technology - User preferences

Current 3D streaming systems are simple add-on to 2D streaming - Poor depth/visual perception and/or discomfort

In Summary …

Customize depth for different displays- Big TV … to phone

Serve different formats- Stereo, V+D, MV + D, …

Adapt to network dynamics - Wired, wireless, …, varying bandwidth

Manage many versions - Compared to 1—3 versions for 2D videos

3D Streaming Systems Need to ..

14

Side By Side

Top Bottom

Frame Sequential

Anaglyph

Row-Interleaved

Column-Interleaved

Video Plus Depth

Display Size

TV

Desktop

Laptop

Tablet

Cell Phone

Display Technology

5 x 7 x 3 = 105 versions for each video

Example

High Quality

Medium Quality

Low Quality

Network Bitrate

15

Enable all devices to render best possible 3D videos

From mobiles to big TVs

Our Challenge

16

Our Solution …

17

18

Show the need for depth optimization - Subjective study on different displays

Anahita: complete system for 3D streaming- Supports most current displays/technologies - Provides efficient management of 3D versions- Enables personalized depth - Is scalable and dynamic (using DASH)

Method for depth expansion and compression- Optimizes depth: from mobile phones to big TVs - Preserves scene structure (important for sports)- Does not introduce visual artifacts- Is computationally inexpensive

Our Contributions

19

Subjective study- 10 subjects- 2 displays (phone & 55” TV)- Series of short 3D clips (from soccer)- Show six different versions

• Original

• Depth compression

• Depth expansion (4 cases)

- Versions are shown in random to subjects- Subjects are asked which version is preferred

The Need for Depth Customization

20

Best version is NOT the original one Depth customization depends on display size

Results

21

SPSS: Structure Preserving Scene Shifting- Expands or compresses depth in stereo video- Simple image processing operations- Preserves lines and planes

- Suitable for field sports: soccer, football, tennis, …

Depth Customization

22

Control disparity of pixels - increase/decrease depth

Two operations- Slant (affects vertical component of disparity) - Stretch (affects horizontal component of disparity)

SPSS: Basic Idea

Depth Gradient

g = (gx, gy)23

Depth Gradient

g = (gx, gy) 24

SPSS: Slant

Left eye image Right eye image

25

gy is changed

SPSS: Slant

26

gx is changed

SPSS: Stretch

27

28

Calculate gradient g = (gx, gy) of scene disparity

Compute slant and stretch factor

Remap left/right views using:

SPSS: Main Steps

Before

After

SPSS: Sample Results

29

Before

After

SPSS: Sample Results

30

31

SPSS applies to scenes with planar depth long shots in field sports Our analysis of multiple full games

- 60—70% of the game

Close-ups and short shots do not benefit from SPSS

Anahita: has automatic shot classifier

SPSS: Coverage

Anahita: System Architecture

32

Anahita: 3D Version Manager

33

34

Based on DASH Versions are organized in segments Most suitable 3D version is chosen for each client

Client can personalize: ask for more/less depth- Served using DASH

Adaptive Streamer

35

Server - Deployed on Amazon- Implements many 3D video processing operations, and

our SPSS depth customization method

Clients- Mobile, tablet (mobile apps)- Stereo TV (web)- Desktop display (web)- Auto stereoscopic 55” display (custom app)

Implementation: Whole System

36

5 displays- mobile, tablet, 15.6” laptop, 27” desktop, 55” TV

3 soccer 3D video clips from YouTube- Man United vs. Wigan (60 sec)- Chelsea vs. Wigan (24 sec)- Chelsea vs. Plymouth (20 sec)

15 subjects- viewed all clips (original & optimized) on all displays in

random- Ranked depth from 1-poor to 5-Excellent

Evaluation: Subjective Study

37

Significant improvements in all cases- up to 35%

Sample Results

38

3D streaming systems are more complex than 2D

Showed the need for depth adaptation

Proposed method to customize depth

Designed 3D streaming system

Evaluation study shows large gains

Conclusions

Anahita: Meaning

• Ancient Persia: Anahita is source of all water, where warm and clear streams flow through golden channels to all lands and oceans on Earth 

• Internet age: Anahita is source of all 3D videos, where high-quality streams of 3D videos flow through network channels to all types of displays