Institute of Computer ScienceChair of Communication Networks

Prof. Dr.-Ing. P. Tran-Gia

Follow the Crowd: On QoE for Internet Applications

Tobias Hoßfeldwww3.informatik.uni-wuerzburg.de

www.t-hossfeld.de

Tobias Hossfeld2

2

What is the Internet crowd consuming?

Global Consumer Internet Traffic Volume (Forecast).Source: Cisco VNI 2011.

2010 2011 2012 2013 2014 20151

2

3

4

5

6

7

8

year

traffi

c in

crea

se (n

orm

aliz

ed to

201

0)

Internet videoOnline gamingWeb, email, and dataFile sharingVoIP

2010 2011 2012 2013 2014 20150

10

20

30

40

50

60

year

Exa

byte

s pe

r Mon

th

Internet videoOnline gamingWeb, email, and dataFile sharingVoIP

Web and Cloud Applications Online Video, Web Browsing, Downloads, Cloud Services, etc.

Why relevant? Constitute dominant internet use cases Generate relevant share of network traffic

Tobias Hossfeld3

3

YOUTUBE QOE AND PRACTICAL GUIDELINES

Tobias Hossfeld4

4

QoE Issue: Waiting, Waiting, Waiting…

Stalling

Tobias Hossfeld5

5

Video Transmission over the Internet

HTTP streaming• Reliable transmission• Video quality not affected• But stalling may occur• Most stimuli/impairments

are of temporal nature

UDP-based streaming• Unreliable transmission• Video quality affected• Artifacts may occur• Stimuli are visual degradations

or artifacts

• YouTube uses HTTP streaming• Internet technology changes quality perception

Tobias Hossfeld6

6

Key Influence Factors on YouTube QoE

Interesting: no significantcorrelation of QoE and initial delay video characteristics like

resolution, type of content,ratio of audio/video, etc.

users preference, whether they liked video

demographical features

Stalling frequency andstalling duration determinethe user perceived quality

Support vector machines and correlation coefficients

Tobias Hossfeld7

7

What is the influence of stalling on YouTube QoE?

Small number of interruptions strongly affect YouTube QoE Provider (i.e. content and network provider) must avoid stalling

Total stalling time not sufficient for good QoE estimation Monitoring of QoE requires sophisticated methods to capture

stalling pattern, e.g. using DPI or directly at end user

Tobias Hossfeld8

8

Provider: Between the Devil and the Deep Blue Sea?

In case of insufficient resources, „one“ has to choose between

initial delays and stalling What is worse for users? Stalling has to be avoided,

even at costs of initial delays

Exponential increase of costs wrt. quantile (of video corpus)

Delivering videos with about 120% of video bitrate as “rule of thumb”

Current work: Is YouTube QoE management beneficial for ISPs? • Users do „QoE management“

themselves – by pausing the video to prefetch contents and then to consume w/o interruptions

• ISP may „invest“ in capacity, sophisticated traffic management, e.g. DASH and SVC

Tobias Hossfeld9

9

CROWDSOURCING FOR QOE TESTING

Tobias Hossfeld10

10

Crowdsourcing

Crowdsourcing is a neologism composed of “crowd“ and “outsourcing“ literally, it means outsourcing to a (large, anonymous) crowd

All tasks are web-based micro jobs, typically little effort to fulfill

Crowdsourcing interesting for (QoE) user studies large user panel, diversity of users, international users, user studies can be executed in short time, low costs in contrast to laboratory studies, QoE tests for Internet applications with realistic settings

Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.

Jeff Howe - Definition of Crowdsourcing “The White Paper Version“

Tobias Hossfeld11

11

Crowdsourcing Workflow

Challenges due to remote setting Unreliable QoE results, no test moderator Heterogeneous environment, devices, users

Employer Worker

Submit task Pull task

Complete task

Remuneration

Crowdsourcing platform

1 2

34

5

Tobias Hossfeld12

12

Countermeasures: Unreliability

Proper Test Design and statistical methods for filtering data Consistency Tests

– “Same” question is asked multiple times in different manner.– Example: user is asked about his origin country in the beginning and about

his origin continent at the end. Content Questions

– Simple questions about the video clip, after watching the video.– Example, “Which sport was shown in the clip?

A) Tennis. B) Soccer. C) Skiing.‘” Application Usage Monitoring

– Example: measuring the time the worker spends on the task – Example: monitor browser events and user reactions

Utilize features of crowdsourcing platform Specialized crowds, which have certain skills, reliability, etc. Conduct training sessions, two-stage tests Payment according to quality

Tobias Hossfeld13

13

Lessons Learned: Unreliable workersFILTER LEVEL 1:- wrong answers to content questions- different answers to the same questions- always selected same option- consistency questions:

specified the wrong country/continent

FILTER LEVEL 2:- did not notice stalling- perceived non-existent stalling

FILTERLEVEL 3:- did not watch all videos completely

Mw1 Mw2Mw3 Mw4 Mw5Mw6 Mw7 FB.0

20

40

60

80

100

Per

cent

age

of w

orke

rs in

test

Level 0 Level 1 Level 2 Level 3

C1 C2 C3 C4 C5 C6 C7 Facebook

• Many user ratings rejected use simple test instructions avoid Java applets take care of low Internet speed avoid incentives for users to cheat,

see Facebook results of student’s friends

• User warning („Test not done carefully“) rejection rate decreased about 50%

• improvements possible detailed analysis of (inter and intra-) rater reliability revealed: filtering too strict

First crowdsourcing tests

Tobias Hossfeld14

14

Crowdsourcing vs. Laboratory Studies

0 1 2 3 4 5 61

2

3

4

5

number of stallings

MO

S

crowdsourcinglaboratory

single stall event: 4 secvideo duration: 30 sec

Crowdsourcing tests with Microworkers.com at Uni Würzburg Lab studies within ACE 2.0 at FTW’s i:Lab Similar results in laboratory and crowdsourcing study Crowdsourcing appropriate for QoE tests of Internet apps

2,035 users from more than 60 countries participated in tests and rated 8,163 video. Payment was below 200,- Euro. User diversity Statistically significant results Low costs, fast conduction

Tobias Hossfeld15

15

CrowdsourcingTests for HD Live Streaming

Live video streaming investigated via Microworkers and Facebook Joint work within QUALINET STSM by Bruno Gardlo “Improving

Reliability for Crowdsourcing-Based QoE Testing”

Strong differences due to worse viewing conditions and smaller screen resolutions context monitoring required, e.g. light conditions,

Critical, proper analysis of data, consider hidden influence factors

Tobias Hossfeld16

16

CURRENT ACTIVITIES IN QUALINET

Tobias Hossfeld17

Qualinet “Crowdsourcing“ Task Force

Goal Derive a methodology and setup for crowdsourcing in QoE assessment, Challenge crowdsourcing QoE assessment approach with usual “lab” methodologies,

comparison of QoE tests Develop mechanisms and statistical approaches for identifying reliable ratings from

remote crowdsourcing users, Define requirements onto crowdsourcing platforms for improved QoE assessment.

Experiences with crowdsourcing What are the main challenges? Reliability, environment/context monitoring, technical

implementation, language problems … Cartography for crowdsourcing use cases and mechanisms Database with crowdsourcing results, e.g. impact of context factors on QoE, country,

habits, …

Framework for crowdsourcing QoE tests Results are implemented in framework “QualityCrowd” by TU Munich Further information:

https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:crowd

Tobias Hossfeld18

18

Desktop

EyeOS

Degree of Interactivity

Small High

Qualinet Task Force „Web and Cloud Apps“

E-Mail

Google Mail

Customer Relationship Management

SalesForce.com

Office

MS Office Live

Appl

icati

on T

ype

Exam

ple

Gaming

OnLive

Instant Messaging

Facebook Chat

• Technology change and service migration to clouds strongly impacts user perception and QoE

• Current activites• Dropbox QoE and mulicollaboration tools• QoE-aware adaptation mechanism for video streaming: DASH

and SVC• Standardization: finalization of model and measurement

methodology for web browsing QoE

• https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:webcloud

Institute of Computer ScienceChair of Communication Networks

Prof. Dr.-Ing. P. Tran-Gia

?

?

Questions?

www.t-hossfeld.dehossfeld@informatik.uni-wuerzburg.de