Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the...
Transcript of Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the...
![Page 1: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/1.jpg)
TechnischeUniversität Berlin URL: www.tkn.tu-berlin.de
Predicting the Perceptual Service QualityUsing a Trace of VoIP Packets
C. Hoene, S. Wiethölter, A. WoliszSeptember 29th, 2004.
Qofis’04, Barcelona, Spain
![Page 2: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/2.jpg)
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
![Page 3: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/3.jpg)
VoIP transmission of a telephone call
![Page 4: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/4.jpg)
How to Judge the Quality of a Call?
Measure the quality of a VoIP system:Gather packet loss rates, mean delay, etc.
Easy but inaccurate
In the end the service quality is importantHuman based listening tests are extensive
ITU P.862 (PESQ algorithm) measures speech qualityCompares original sample with the transmitted versioncalculates Mean Option Score (MOS) (1=bad, 5=excellent)
ITU G.107 (E-Model) predicts quality of tel.-systemConsiders echo, loudness, coding, packet loss rate, delay, …Result: R Factor (0=bad, 70=toll quality, 100=excellent)
![Page 5: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/5.jpg)
Limits of PESQ and E-Model
PESQ and E-Model cannot judge the impact of:Impact of playout scheduling cannot be predicted.Variable packet delay
adaptive playout timeNon-random packet losses Not applicable to assess VoIP packet traces resulting from simulations or experiments
![Page 6: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/6.jpg)
Predicting Playout Scheduling
A VoIP phone can implement any playout scheduler. How to know which is going to be used? Actually, you cannot know it.But you know the most common, published playout scheduler schemes, which are
Fixed-deadline and adaptive from Van Jacobsen, Mills, Schulzrinne, Ramjee, and Moon
Calculate packets' playout times and the mean transmission delay.
Consider only speech frames during voice activity because during silence a human cannot identify the delay.
We can predict the behavior of most phones !
![Page 7: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/7.jpg)
Assessing a Trace of VoIP Packets
Our solution:Using the most common different playout schedulers and encoding schemescombine of E-Model and PESQ (MOS to R formula approved by the ITU)
PESQ calculate speech qualityE-Model combinesMOS rating and transmission delays
![Page 8: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/8.jpg)
Speech Quality and Transmission Delay
Speech Recordings
SpeechCoding
(coding distortion)
Experimentor Simulation
(loss or delay)
Decoder PLC
PESQReferenceDegraded
Audio
AudioMOS
Playout Buffer
(loss or scheduling)
E-ModelIe Result:Id R Factor
Degraded Audio
Mean Delay
![Page 9: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/9.jpg)
Equations – referring to the SPECTS’04 paper…
![Page 10: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/10.jpg)
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
![Page 11: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/11.jpg)
PESQ not verified conduct listening tests
Can PESQ measure playout rescheduling?or non-random packet losses?
PESQ can measure variable playout but its performance has never been verified.It has been tested only for random/burstybut not for non-random losses.
Conduct formal listening-only tests!
![Page 12: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/12.jpg)
Test Sample Design
Construct packet traceswith delay spikes!Delay spikes with a height of 50 to 300ms and a width of height+10%.1 Spike in one sample
What to do?1. Drop late packets2. Or delay playout?
![Page 13: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/13.jpg)
Playout Decisions
Original
a) Packet loss b) Positive adaptation c) Negative adaptation
1 2 2+ 2++ 5 6 1 2 2+ 2++ 3 4 5 6 1 2 5 6
Legend:1...6 Frame seq. no. 2+;2++ Concealed frame
1 2 3 4 5 6
1. DROP: Drop late packets (a)2. ADAPT: Delay playout to late packets (b)3. ADAPT&FALLBACK: Delay playout (b) and return to
normal during the next silence period (c)
![Page 14: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/14.jpg)
Listening-Only Tests
Follow ITU P.830 butMultiple person at the same timeUse native and foreign speakers Use linear MOS scale (no discrete scale!)Sample Sequence: For each sentence
1. Original at the beginning, 2. MNRU with 5dB (worst),15,25,35 (best)3. Randomly distributed: packet loss samples
Produced on CD containing all the samples
![Page 15: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/15.jpg)
Studio...
![Page 16: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/16.jpg)
Listening-Test and Sample DesignDifferent encoding schemes:
G.711, G729
Produced delay spikes (only during voice activity)with length 50,100,150,200,250,300ms with width=length*1.1
Different samples with different length 5-10s), German (using Kiel Corpus Vol. I)220 samples (including MNRU samples as a reference)
Coding Loss Generator
DecodingPLC
Listening-only Tests
Frame Analysis PESQ
reference
speechproperties
importancePESQ MOS
X
seed
Speech Recordings R
(Correlation)
pack
etiz
atio
n
loss
rate
rate
/mod
e
algo
rithm
sam
ple
spea
ker
lang
uage
MOS
![Page 17: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/17.jpg)
Rating Performance
Three parts: intro, first and second half. First is best.Native speaker and foreign speaker rate as good.Some persons did not kept track with the sample no.
![Page 18: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/18.jpg)
Human LQS-MOS vs. PESQ LQO-MOS
y = 0,9403x - 0,2139R= 0,8316
1
2
3
4
1 2 3 4
MOS (Humans)
MO
S (P
ESQ
)
![Page 19: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/19.jpg)
MNRU vs. MOS: Correlation: R=0.977
1
2
3
4
5 15 25 35 45
MNRU
MO
S
MOS PESQ MOS MOS cited scaled MOS
![Page 20: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/20.jpg)
Results for G.711 coding
![Page 21: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/21.jpg)
MOS Variance vs. Prediction Performance
trend line: R=0.768
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1,0
0 0,2 0,4 0,6 0,8 1PESQ sample variance for different impairment patterns
Cor
rela
tion
betw
een
PESQ
and
hum
ans
![Page 22: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/22.jpg)
Content
Introduction/BackgroundOur ApproachPlayout SchedulingListening-only Tests
Delay SpikesNon-Random Packet Losses
Conclusions
![Page 23: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/23.jpg)
Introduction
Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range,depending on
the frame speech propertiesthe encoder/decoder/concealment algorithmsdecoders resynchronization time after loss (especially low-rate decoders might maintain a wrong state after loss for the following frames.)the surrounding speech.
Example: Discontinuous Transmission (DTX)Speech frames during silence are less importantLower frame rate during silence
![Page 24: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/24.jpg)
Packet Importance
The packet’s importance is the quality degradation that its loss would cause.
Definition: The importance of frame losses is the difference between the speech quality due to coding loss and the quality due to coding loss and frame losses, times the length of the analyzed sample.
Conducted some million PESQ ratings Dropped only one VoIP packet in a sample
![Page 25: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/25.jpg)
Just try it: www.tkn.tu-berlin.de/research/mongoliaSelect predefined parameter sets
chose sample
Compression?
Listen to it!
amount of loss
Choose an another loss pattern
Drop only frames with an importance between
min. and max.
Talking or silence? Voiced or unvoiced?
Judge the speech quality by your self!
Sample statistics
![Page 26: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/26.jpg)
![Page 27: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/27.jpg)
![Page 28: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/28.jpg)
![Page 29: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/29.jpg)
![Page 30: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/30.jpg)
Conclusion
Presented an approach to assess VoIP packet tracesCombined PESQ, E-Model and playout scheduling because
Speech frames differ in their importancePlayout scheduler are not standardizedPlayout rescheduling harms speech quality.
Verified PESQ for non-random packet losses (R=0.94) and delay spikes (R=0.87).
Precision performances depend on the variance of the samples’ speech quality.
![Page 31: Predicting the Perceptual Service Quality Using a … Losing one Voice-Over-IP packet impairs the perceptual quality in a wide range, depending on the frame speech properties the encoder/decoder/concealment](https://reader033.fdocuments.us/reader033/viewer/2022052710/5aaca4867f8b9a2b4c8d6f94/html5/thumbnails/31.jpg)
Assessing VoIP: SummaryOpen-source software including
Implementation of all common playout schedulersG.711 and G.729 codingSample database including human rating results
Verified software in various research projects, e.g. voice over WLAN, impact of handover, ad-hoc.http://www.tkn.tu-berlin.de/reseach/qofis
C. Hoene, S. Wiethölter, and A. Wolisz, "Predicting the Perceptual Service Quality Using a Trace of VoIP Packets", In Proceedings of QofIS’04, Barcelona, Spain, September 2004.C. Hoene, H. Karl, and A. Wolisz, "A Perceptual Quality Model for Adaptive VoIP Applications", In Proceedings of International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS'04), San Jose, California, USA, July 2004, Paper won the Best Paper Award of the conference. C. Hoene and E. Dulamsuren-Lalla, "Predicting Performance of PESQ in Case of Single Frame Losses", In Proc. MESAQIN 2004, Prague, CZ, June 2004.S. Möller and C. Hoene, "Information About a New Method For Deriving the Transmission Rating Factor R From MOS in Closed Form", ITU, May 2002, Temporary Document for the study group 12.