QoE Assessment in Olfactory and Haptic Media Transmission: Influence of

Inter-Stream Synchronization Error

Sosuke Hoshino, Yutaka Ishibashi,

Norishige Fukushima, and Shinji Sugawara

Department of Scientific and Engineering Simulation

Nagoya Institute of Technology

Nagoya 466-8555, Japan

Outline

Background Purpose Olfactory and Haptic Media Display System Method of Assessments QoE Assessment Method QoE Assessment Results Conclusions and Future Work

Background

Multi-sensory communications have been emerging.

Olfactory, haptic and visual media

Transmissionover the Internet

Degradation of qualityof experience (QoE)

It is necessary to clarify the influence of inter-stream synchronization error on QoE.

For efficient media synchronization control

We need to carry out mediasynchronization control.

This studyfocuses on these media.

• Network delay • Delay jitter • Packet loss

Vision, auditory, olfaction,

tactile, gustation

No one has investigated influence of inter-stream synchronization error between olfactory and haptic media on QoE.

Purpose

Influence of inter-stream synchronization error between olfactory media and video is investigated on QoE.

(inter-stream synchronization errors between about 30 ms and around +20 ms are hardly perceived).

Previous Work

We deal with harvesting fruit in a 3-D virtual space for an olfactory and haptic media display system.

By QoE assessment, we investigate the influence of inter-stream synchronization error between olfactory and haptic media.

This Work

The haptic media are assumed to be synchronized with the visual media for simplicity.

Olfactory and Haptic Media Display System

• A user can touch and catch a grapefruit by manipulating PHANToM.• The smell of grapefruit is diffused by SyP@D2 when a grapefruit is picked since the picked grapefruit approaches the user’s nose.

Olfactory Display (SyP@D2)

Distance between a user and SyP@D2: About 0.3 m Intensity of diffusion: Maximum

It takes about 2 seconds for a user to perceive the smell.

SyP@D2 displays a smell to a user by blowing air

into a smell cartridge attached to the main body.

Functions of Olfactory and Haptic Media Display System

Every millisecond

Every 10 milliseconds

We assume that olfactory and haptic media are synchronized

when a user starts to perceive a smell on picking a grapefruit.

Smell of grapefruit

Method of Assessments (1 of 4)

A user catches and starts to pull

a grapefruit.

The user releases the grapefruit.

Haptic media

Olfactory media

Reaction force when pulling a grapefruit

Reaction force by weight of grapefruit

: Starting to output the smell

: Outputting the reaction force when picking a grapefruit from a tree

The user’s force exceeds 2.45 N.

Buffering

When we advance output timing of olfactory media

Smell threshold value

When we delay output timing of olfactory media

Method of Assessments (2 of 4)

We start to output the olfactorymedia when the force given by a

user exceeds smell threshold value.

We generate a delay by buffering of the olfactory media.

We generate temporal difference by outputting olfactory media earlier or later than haptic media.

If a user picks grapefruit quickly, it is impossible for smell to reach a user before picking grapefruit even if we use the smell threshold value.

However,

We employ picking time in order to enable the smell to reach the user before picking grapefruit.

Olfactory and haptic media are synchronized when

a user starts to perceive a smell on picking a grapefruit.

Smell of grapefruit

Method of Assessments (3 of 4)

A user catch and start to pull a grapefruit.

The user release the grapefruit.

Haptic media

Olfactory media

Reaction force when pulling grapefruit Reaction force by weight of grapefruit

: Starting to output the smell

: Outputting the reaction force when picking a grapefruit from a tree

The user’s force exceed 2.45 N.

If the user’s force continues to exceed 2.45N for a specified time,

the grapefruit is picked.picking time

Smell threshold valuePicking time

Method of Assessments (4 of 4)

picking time

longshortIt is impossible for smell to reach

a user before picking a grapefruit.

As we increase the picking time, it takes a longer time for a user to pick a grapefruit.

We make two assessments (Assessments 1 and 2).

The feel of picking a grapefruit in our system

is different from that of reality.

Picking time

Assessment 1 0 ms

Assessment 2 1500 ms

Task: Each subject operates PHANToM and picks grapefruits in the virtual space.

Assessment item:

Total assessment time in each of Assessments1 and 2: About half an hour We make Assessment 1 first. Then, we make Assessment 2 on a different day.

QoE Assessment Method (1 of 2)

Inter-stream synchronization qualityWe obtain mean opinion score (MOS).

Score Description

5 Excellent

4 Good

3 Fair

2 Poor

1 Bad

Assessments 1 and 2

Subjects: Number : 20 Age: Between 21 and 30

Subjects pick grapefruits while listening to classical music (The Four Seasons  by Vivaldi).

Reason: To prevent the subject from being influenced by sound of blowing from SyP@D2.

Subjects take a 30-second rest every judgment. Reason: To prevent him/her from being influenced by

lingering smell in the air and human olfactory adaptation.

QoE Assessment Method (2 of 2)

QoE Assessment Results (1 of 5) Inter-stream synchronization quality in Assessment 1 (picking time: 0 ms)

Outputting olfactory media earlier

It takes approximately 2 secondsto perceive smell.

About 1380 ms

About 250 ms

Average of inter-stream synchronization error

Outputting olfactory media later

Constant delay ofolfactory media

The subjects hardly perceive the inter-stream synchronization

error when it takes from about620 ms to around 1750 ms to

perceive the smell after pickinga grapefruit

QoE Assessment Results (2 of 5) Inter-stream synchronization quality in Assessment 1 (picking time: 0 ms)

The inter-stream synchronization error is

allowable when it takes from about 620 ms to around 2400 ms to perceive the smell after

picking a grapefruit.

About 1380 ms

About +400 ms

We cannot assess the inter-stream synchronization quality when

the inter-stream synchronization error is smaller than about 1380 ms.

It takes approximately 2 secondsto perceive smell.

Outputting olfactory media earlier

Outputting olfactory media later

QoE Assessment Results (3 of 5) Inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms)

The subjects hardly perceive the inter-stream

synchronization error when it takes from about 500 ms to

around 1000 ms to perceive the smell after picking a

grapefruit.

About 1500 ms Around 1000 ms

Outputting olfactory media earlier

It takes approximately 2 secondsto perceive smell.

Outputting olfactory media later

Constant delay ofolfactory media

Average of inter-stream synchronization error

Inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms)

QoE Assessment Results (4 of 5)

The inter-stream synchronization error is

allowable when it takes from about 110 ms to around 1400 ms to perceive the smell after

picking a grapefruit.About 1890ms

Outputting olfactory media earlier

It takes approximately 2 secondsto perceive smell.

Outputting olfactory media later

Around 600ms

QoE Assessment Results (5 of 5)

Comparison between Assessments 1 (picking time: 0 ms) and 2 (picking time: 1500 ms)We adopt results of Assessment 2 for

safety.

To clarify the reason ofthis result is for further study.

Outputting olfactory media earlier

The MOS value of the inter-stream synchronization quality in Assessment 2 is smaller than

that in Assessment 1.

Outputting olfactory media later

Conclusions

When it takes from about 500 ms to around 1000 ms to perceive the smell after picking a grapefruit,

the inter-stream synchronization error is hardly perceived. When it takes from about 110 ms to around 1400 ms to

perceive the smell after picking a grapefruit, the Inter-stream synchronization error is allowable.

The difference in the feel of picking between two picking times is not large.

We investigated the influence of inter-stream synchronization error between olfactory and haptic media on QoE.

We also examine which of picking times (0 ms and 1500 ms) is felt more real.

Future Work

We plan to clarify the reason why the MOS value of the inter-stream synchronization quality in Assessment 2 (picking time: 1500 ms) is smaller than that in Assessment 1 (picking time: 0 ms).

We will investigate the influence of inter-stream synchronization error between olfactory and haptic media when the diffusion of the smell stops after harvesting a grapefruit.

We will examine the influences of packet loss and network delay jitter on QoE.

We will study media synchronization control which takes advantage of the results obtained in this study.

Olfactory display (SyP@D2)

We can attach six cartridges to SyP@D2. SyP@D2 can diffuse each smell at any time. There are some parameters of SyP@D2 which we

can set up. Name of smell, Duration of diffusion, Intensity of

diffusion Maximum wind speed of SyP@D2:

approximately 0.15 m/sWe set the distance between a user and

SyP@D2 to about 0.3 m. We measured the average time that it takes for the user to perceive a smell.

Task: Each subject operates PHANToM and picks grapefruits in the virtual space.

Assessment item:

Total assessment time: About half an hour Subjects:

Number : 20 Age: Between 21 and 30

QoE Assessment Method

Inter-stream synchronization qualityWe obtain MOS. Score Description

5 Excellent

4 Good

3 Fair

2 Poor

1 Bad

We assume that olfactory and haptic media are synchronized

when a user ceases to perceive a smell on releasing a grapefruit

(if the user release the grapefruit, the grapefruit disappears).

Inter-stream synchronization quality

QoE Assessment Results (when the diffusion of the smell stops after harvesting a grapefruit)

Outputting olfactory media earlier

Outputting olfactory media later

Inter-stream synchronization error [ms]

About 1310 ms

About 960 ms About 520 ms

About +400 ms

I 95% confidence interval

Comparison between Assessments 1 (picking time: 0 ms) and 2 (picking time: 1500 ms)

QoE Assessment Results

The allowable range in this study is narrower than that in previous study.

This is because we need higher interactivity in the case where the olfactory and haptic media are perceived together than in the case

where the olfactory media and video are perceived together.

When the inter-stream synchronization error is between about 30 seconds and around +20 seconds, the inter-stream synchronization

error between olfactory media and video is allowable.

　

QoE Assessment Results (haptic and visual media)

Inter-stream synchronization error [ms]

I 95% confidence interval

Inter-stream synchronization quality when we change the haptic media

QoE Assessment Results (haptic and visual media)

Inter-stream synchronization error [ms]

I 95% confidence interval

Inter-stream synchronization quality when we change visual media

Method of Assessments (5 of 5)

Picking time Inter-stream synchronization error

Assessment1

0 ms

About 1380 ~ around 90 ms

0 ~ +1000 ms (intervals of 100ms)

+1000 ~ +3000 ms (intervals of 500ms)

Assessment2

1500 ms

About 2680 ~ around 1740 ms

1500 ~ 500 ms (intervals of 100ms)

500 ~ +1500 ms (intervals of 500ms)

We change the smell threshold value from 0 N to 2.10 N at intervals of 0.35 N in order to alter the average of

the inter-stream synchronization error.

We also investigate difference in feel of picking a grapefruit.

Task: Subjects pick some grapefruits for each value of picking times (0 ms and 1500 ms) and are asked to choose one from among three options.

Total assessment time of this assessment: Around three minutes

We make this assessment immediately after carrying out Assessment 2.

1: Feel in the case where picking time is set to 0 ms is better.2: Feel in the case where picking time is set to 1500 ms is better.3: There is almost no difference between the two cases.

QoE Assessment Method (2 of 3)

Difference in Feel of Picking between Two Picking Times

The difference in the feel of picking between two picking times is not large.