Quality of Multimedia Experience Past, Present and Future

ACM MultimediaOctober 22nd, 2009, Beijing, China

Quality of Multimedia Experience

Past, Present and Future

Prof. Dr. Touradj Ebrahimi

Touradj.Ebrahimi@epfl.ch

What is “quality” of multimedia content? How is multimedia content “quality” measured

today? What are trends in assessment of “quality” in

multimedia? What are the challenges ahead?

Today we will talk about…

Quality – a simple yet difficult concept

Like many human sensations quality is easy to understand but difficult to define

According to Wikipedia:– A quality (from Latin - qualitas) is an attribute or a

property. – Some philosophers assert that a quality cannot be

defined. – In contemporary philosophy, the idea of qualities and

especially how to distinguish certain kinds of qualities from one another remains controversial.

A fundamental yet largely under-investigated concept

Aristotle classified every object of human apprehension into 10 Categories– Substance, Quantity,

Quality, Relation, Place, Time, Position, State, Action, Affection

Aristotle 384 BC – 322 BC

Some definitions according to dictionary

Definition 1– General : Measure of excellence or state of

being free from defects, deficiencies, and significant variations.

– ISO 8402-1986 standard defines quality as "the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs"

Definition 2– Manufacturing : Strict and consistent

adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements.

Definition 3– Objective : Measurable and verifiable aspect

of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness.

Definition 4– Subjective : Attribute, characteristic, or

property of a thing or phenomenon that can be observed and interpreted, and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste.

Definition according to ISO 9000

ISO 9000: a family of standards for quality management systems

Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements– High quality: if characteristics meet requirements– Low quality: if characteristics do not meet all

requirements

Quality is a relative concept– Degree of quality

Quality – is in fact an elephant

The blind men and the elephant: Poem by John Godfrey Saxe

Quality of Service in computer networks and communications

Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results

Quality in QoS framework

Network QualityCapacity

CoverageHandoff

Link QualityBitrate

Frame/Bit/Packet lossDelay

User QualitySpeech fidelity

Audio fidelityImage fidelityVideo fidelity

Quality of Service in computer networks and communications

Quality of Service (QoS)– Resource reservation control mechanisms– Ability to provide different priority to different

applications, users, or data flows– Guarantee a certain level of performance

(quality) to a data flow

(Service) Provide-centric concept Tightly related to the concept of Mean

Opinion Score (MOS)

What is Mean Opinion Score (MOS)?

Widely used in many fields:– Politics/Elections– Marketing/Advertisement– Food industry– Multimedia– …

The likely level of satisfaction of a service or product as appreciated by an average user

Should be performed such that it generates reliable and reproducible results– Subjective evaluation methodology– More complex and difficult that it a priori seems

What is behind a MOS?

A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:

Subjective evaluation

• Test/lab environment• Test material• Test methodology• Analysis of the data

Test/lab environment

Type of Monitors/Speakers and other test equipments Lighting /Acoustic conditions Laboratory architecture, background, … Viewing distance /Hearing position …

Test material

Meaningful content for the envisaged scenario/application– Typical content– Worst case content– …

p01 p06 p10 bike cafe woman

Test methodology

Subjects– Naïve or Expert?

Instructions– Which questions to ask subjects and how– Training

Presentation– Single or double stimulus– Sequential or simultaneous

Grading scale– Numerical– Categorical

• Test conditions and methodologies are specified in:

Recommendation ITU-R BT. 500-11 “Methodology for the subjective assessment of the quality of television pictures” (1974-2002).

Recommendation ITU-T P. 910 “Subjective video quality assessment methods for multimedia applications” (1999).

Recommendation ITU-R BT. 1788 “Methodology for the subjective assessment of video quality in multimedia applications” (2007).

Based on television scenario!

ITU Recommendations for test methodologies

Single Stimulus (SS)

Test methodology (I)

Non-categorical adjectival or numerical grading scale

5 Excellent 4 Good

3 Fair

2 Poor

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying

Excellent

Categorical adjectival grading scale: Categorical numerical grading scale:

“Rate from 1 to 11”

Double Stimulus Impairment Scale (DSIS)

Test methodology (II)

Categorical impairment grading scale:

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying

Double Stimulus Continuous Quality Scale (DSCQS)

Test methodology (III)

Sample 1Sample 2

Non-categorical adjectival or numerical grading scale:

Excellent

Sample 1 Sample 2

Excellent

Stimulus Comparison (SC)

Test methodology (IV)

Categorical adjectival comparison scale:

“same or different” much worse

slightly worse

the same

slightly better

better

much better

Non-categorical judgement:

Much worse

Much better

Single Stimulus Continuous Quality Evaluation (SSCQE)

Test methodology (V)

(Very annoying)

(Imperceptible)

Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)

Test methodology (VI)

(Much better)

(Much worse)(Reference) (Test sequence)

Analysis of the data

• Scores distributions across subjects is assumed to be close to normal distribution

• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals

NNtCI j

),2/1(

mij = score by subject i for test condition j.

N = number of subjects after outliers removal.

t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 degrees of freedom and a desired significance level α (α=0.05 in our case).

σj = standard deviation of the scores distribution across subjects for test condition j.

What is behind a MOS?

Relationship between estimated mean values

• Hypothesis test to find out whether the difference between two MOS values are statistically significant

Two-sided t-test:

• t statistic:

• Decision rule to reject H0:

BA MOSMOSH :0

BAa MOSMOSH :

MOSMOSt

BAobs 22

),2/1(),2/( NttORNtt obsobs

MOS hypothesis testJPEG 2000 4:2:0

JPEG 2000 4:4:4

JPEG XR MS

JPEG XR PS

JPEG 2000 4:2:0

JPEG 2000 4:4:4

JPEG XR MS

JPEG XR PS

0.25 bpp

0.50 bpp

0.75 bpp

1.00 bpp

1.25 bpp

1.50 bpp

0Number of times

H0 is rejected

• Subjective tests are time consuming, expensive, and difficult to design

• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics• No reference metrics• Reduced reference metrics

Objective quality metrics

FR, RR and NR scenarios Full Reference approach:

Reduced Reference approach:

No Reference approach:

Input/Reference signal

Output/Processed signal

signalprocessing

FR METRIC

signalprocessing

NR METRIC

signalprocessing

Features extraction

RR METRIC

Full Reference scenario Metrics which look at the fidelity of the signal when compared to an

explicit reference:

processed signal=

perfect quality reference signal+

error signal

MOS predictors based on fidelity measures

Examples Mean Square Error (MSE) Peak Signal to Noise Ratio (PSNR) Maximum Pixel Deviation (Linf) Weighted PSNR Masked PSNR Structural SIMilarity (SSIM) Multiscale Structural Similarity (MSSIM) Visual Information Fidelity (VIF) etc…

MOS predictors based on fidelity metrics

Peak Signal to Noise Ratio

Widely used because of its simplicity and ease in formalizing optimization problems!

For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation

MSE)12(

log10PSNR2B

2ba y)](x,Imy)(x,[Im

where:

M, N = image dimensions Ima , Imb = pictures to compare

B= bit depth

PSNR for color images/video (I)

Several alternatives to compute PSNR for color images/video:

WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3

)MSEwMSEwMSE(w

1)(210log

332211

WPSNR_MSE

WPSNR_PIX

xbbbaaa

)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(ImwMN

2332211332211

10 112

PSNR for color images/video (II)

Which color space to use? RGB Y’CbCr other?

Which weights to use? w1=w2=w3=1/3 w1=0.8, w2=w3=0.1 other?

PSNR for color images/video (III)

on R component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

on G component: on B component:

on Y’ component: on Cb component: on Cr component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

PSNR for color images/video (IV)

Human Visual System (HVS) based metrics simulating properties of the early stages of the HVS Examples: PSNR-HVS-M, etc… simulating high level features of the HVS Examples: Osberger’s metric, etc…

Better correlation with human perception. High complexity.

MOS predictors based on fidelity metrics

Computationalmodel of thevisual system

Visibility Map

PSNR-HVS-M

10 MSE)12(

log10MHVSPSNR

1ncijH )n,m(T)n,m(XKMSE

where: M, N = image dimensions K = constant

= visible difference between DCT coefficients of the original and distorted based on a contrast masking

Tc = matrix of correcting factors based on standard visually optimized

JPEG quantization tables B= bit depth

ij)n,m(X

Metrics based on the hypothesis that the HVS is highly adapted for extraction of structural information from the content of a still image or video. degradation of still images or video = perceived structural information

variation Structural Similarity by Wang et al.

MOS predictors based on fidelity measures

Mean SSIM (MSSIM)

12121 C

C2)Im,(Iml

22121 C

C2)Im,(Imc

SSIM(Im1,Im2) = l(Im1,Im2)[ ]αc(Im1,Im2)[ ]

βs(Im1,Im2)[ ]

γ)0,0,0(

• Luminance comparison function: (C1=constant)

• Contrast comparison function: (C2=constant)

• Structure comparison function:

32,121 C

C)Im,(Ims

(C3=constant)

MSSIM vs PSNR

PSNR = 24.9 dBfor all the images

Mean SSIM (MSSIM)

MSSIM=0.9168 MSSIM=0.6949MSSIM=0.7052

At times one is interested in specific types of distortions that occur in multimedia systems

Examples Bluriness Blockiness Ringing/Mosquito noise Jerkiness etc…

Specific distortion metrics

Blur metric

A perceptual quality blur metric without a reference image. Example:

Gaussian blurred image JPEG2000 compressed image

NR blur metric

140 145 150 155 160 165 170 175 1800

P2 P1 P2' P4 P3 P4'

Pixel position

Pixel value

Correlation subjective ratings / NR blur metrics

4 5 6 7 8 9 10 11 12 13 140

Objective blur measure

Gaussian blurred images

4 5 6 7 8 9 100

Objective blur measure

JPEG 2000 images

96% correlation 85% correlation

Multimedia communication – a definition

Multimedia is about sharing experience (real or imaginary) with others

In a way it all started with story telling and wall drawing around the fire in the caves of early men

Modern multimedia systems are evolved versions of the good old story telling and wall drawing, which hopefully offer increasingly richer experience

The degree of richness of the experience is measured by Quality of Experience (QoE)

Evolutions versus Revolutions in multimedia

Evolution: A given modality improves itself in terms of Quality of Experience:– B&W TV– Color TV– Stereo and CD quality audio TV– HDTV

Revolution: New modalities and expression are introduced bringing new dimensions in Quality of Experience– Photography– Cinema– Internet

Some of the major milestones in multimedia

Story telling and cave drawing Books and written press Photography Telegraph Telephone Radio and music recording Cinema Television and video recording Internet (including VOIP, IPTV, etc.) Mobile communication Social networking (Web 2.0) (What is next?)

Quality of Service vs Quality of Experience

Quality of Service: Value of the average user’s experience richness estimated by a service/product/content provider

Quality of Experience: Value (estimated or actually measured) of a specific user’s experience richness

Quality of Experience is the dual (and extended) view of QoS problem

– QoS=provider-centric– QoE=user-centric

Factors impacting Quality of Experience

Context Context

Quality of Experience in networked multimedia

Quality Wheel

Trends in QoE

Digital world has (re-)discovered the notion of quality– Lower quality content is less and less tolerated by

end-users– Digital technology can now rival and even surpass

the old analog systems performance while remaining cost effective

Increasing interest in QoE– Extending from device-centric and system-centric

quality optimization to end-to-end and especially user-centric optimization

Trends in QoE community building

Increased interest in workshops and conferences around the notion of quality assessment and metrics– QoMEX: International Workshop on Quality of Multimedia

Experience (http://www.qomex.org)– VPQM: International Workshop on Video Processing and

Quality Metrics for Consumer Electronics (http://www.vpqm.org)– IMQA: International Workshop on Image Media Quality and its

Applications (http://www.mi.tj.chiba-u.jp/imqa2008/)

QoE is one of the issues referred to in future funding programs by the EC– Workshop on “The Future of Networked Immersive Media”

(http://cordis.europa.eu/fp7/ict/netmedia/workshop/ws20090610_en.html)

Trends in standardization

Standardization efforts in quality assessment and metrics– Video Quality Experts Group (VQEG)– ITU-T SG 12 (Performance, QoS and QoE)– ITU-R WP6C (Prog. production and quality assessment)– JPEG (Advanced Image Coding - AIC)– MPEG (High performance Video Coding – HVC)– …

Challenges ahead

Some key issues in QoE need to be better addressed– Content-dependent quality assessment methods and metrics – Context-dependent quality assessment methods and metrics– Quality assessment methods and metrics beyond AV (haptics, …)– Multi-modal quality assessment methods and metrics (AV, …)– 3D quality assessment methods and metrics (3D sound, 3D video, …)– HDR content quality assessment methods and metrics – Interaction quality metrics (closely related to usability)– Presence/immersion quality metrics– …

Need for Quality Certification Mechanisms of multimedia services and products– Similar in idea to ISO 9000 series

What does this all mean to you?

Era of user-centric multimedia has already started– It is not anymore sufficient to merely add new

features and functionalities to multimedia systems

– True added value in terms of impact on user’s experience of such features and functions should be demonstrated

– Quality of Experience plays a central role in this new game

Thank you for your attentionQuestions?

Acknowledgements

Some of the concepts and illustrations presented in this talk are the results of collaborations with individuals with whom I have had the pleasure of working. In particular:

– Francesca de Simone (EPFL/MMSPG)– Frederic Dufaux (EPFL/MMSPG)– Lutz Goldmann (EPFL/MMSPG)– Jong-Seok Lee (EPFL/MMSPG)– Andrew Perkis (NTNU/Q2S)– Ulrich Hoffmann (NTNU/Q2S)– Fitri Rahayu (NTNU/Q2S)

Part of the work presented here are the fruits of the research projects:

– EC funded NoE Petamedia– Swiss NSF funded NCCR IM2

Quality of Multimedia Experience Past, Present and Future

Documents

Transcript of Quality of Multimedia Experience Past, Present and Future

Medical Education in Thailand: Past Present andThailand ... · PDF fileThailand: Past Present andThailand: Past, Present ... zClinical sciences examination (MCQ) ... the-jobtrainingprogramjob

Present Perfectnad Past

Past,present,andfutureofsimultaneouslocalizationandmapping ...

Past or present

Past Present Future

Irregular Verbs Present Past Past Particlple

Past vs. Present

Multimedia and the Future of Knowledge - HP Labs · • Opportunities and responsibilities of multimedia experts to craft the past, present and future of content in a way which both

PAST \ PRESENT

Present Past Present Continuous Tense

Past, present - ku

Jeopardy Present Past

Digital multimedia audio forensics: past, present …fac.ksu.edu.sa/.../digital_multimedia_audio_forensics.pdfof tampering. Efforts have been made to detect post-processing such as

Past & present perfect

Licensed to - UCSB · CHAPTER 1 Introduction to Multimedia— Past, Present, and Future The definition of the word multimedia has gone through a large number of evolutionary steps

Educational Multimedia Databases: Past and Present · Chapter 2 Educational multimedia systems and projects Please note that most of the information below is cited or adapted from

Madrid past & present

Hotel past & present

Wkh E tiWorkshop on Energetics - Past and Present - Past ... · Past and Present Michael Pecht ... Wkh E tiWorkshop on Energetics - Past and Present - ... propellants, firecrackers,

past present invention.ppt