Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

REAL-TIME, NON-INTRUSIVE EVALUATION OF

VOIPUSING GENETIC PROGRAMMING

A. Raja1 A. Azad2 C. Flanagan1 C. Ryan2

1Wireless Access Research CentreDepartment of Electronic and Computer Engineering

2Bio-Computing and Developmental SystemsDepartment of Computer Science and Information Sysmtems

University of Limerick, Limerick, Ireland

EuroGP 2007 – 10th European conference on GeneticProgramming

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OUTLINE

1 MOTIVATIONPreambleThe Problem of Speech Quality AssessmentVoice Over IPResearch Goal

2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics

3 GP EXPERIMENTS

4 TEST RESULTS

5 CONCLUSIONS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OUTLINE

1 MOTIVATIONPreambleThe Problem of Speech Quality AssessmentVoice Over IPResearch Goal

2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics

3 GP EXPERIMENTS

4 TEST RESULTS

5 CONCLUSIONS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OUTLINE

1 MOTIVATIONPreambleThe Problem of Speech Quality AssessmentVoice Over IPResearch Goal

2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics

3 GP EXPERIMENTS

4 TEST RESULTS

5 CONCLUSIONS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OUTLINE

1 MOTIVATIONPreambleThe Problem of Speech Quality AssessmentVoice Over IPResearch Goal

2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics

3 GP EXPERIMENTS

4 TEST RESULTS

5 CONCLUSIONS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OUTLINE

1 MOTIVATIONPreambleThe Problem of Speech Quality AssessmentVoice Over IPResearch Goal

2 VOIP SIMULATION ENVIRONMENTSimulation SystemNetwork Traffic Characteristics

3 GP EXPERIMENTS

4 TEST RESULTS

5 CONCLUSIONS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

PREAMBLE

VoIP – A paradigm shiftBandwidth redundancy exploitationQoS remains dominated by network/transport layerdegradationsQuality assessment ...Reflects upon the operating conditions of the network

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

SPEECH QUALITY ASSESSMENT METHODOLOGIES

Two approaches to speech quality Assessment1 Subjective Assessment2 Objective Assessment

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

SUBJECTIVE ASSESSMENT OF SPEECH QUALITY

Speech quality is estimated by humans.Advantage – Reliable results.Limitations

1 Expensive2 Time Consuming3 Laborious4 Lack of Repeatability

Mean Opinion Score (MOS) is the measure of quality.1 – bad5 – Excellent

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITY

A computer automated fast and reliable program is used toassay human perception of speech qualityTwo approaches:

1 Intrusive Assessment2 Non-Intrusive Assessment

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYINTRUSIVE ASSESSMENT

The signal under test is compared against a correspondingreference signal.Advantages:

1 The most reliable artificial means of estimating speechquality

2 Tests can be repeated easilyLimitations:

1 Consumes considerable computing resources.2 Is not useful for continuous monitoring of quality due to

requirement of a reference signal.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYINTRUSIVE ASSESSMENT

The signal under test is compared against a correspondingreference signal.Advantages:

1 The most reliable artificial means of estimating speechquality

2 Tests can be repeated easilyLimitations:

1 Consumes considerable computing resources.2 Is not useful for continuous monitoring of quality due to

requirement of a reference signal.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYITU-T P.862 (PESQ)

PESQ algorithm is the current ITU-T Recommendation forintrusive speech quality estimation.The speech signal is mapped from time domain totime-frequency representation using the psychophysicalequivalents of frequency and intensity.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYITU-T P.862 (PESQ)

It has shown a high correlation with various ITU-Tbenchmark tests.For 30 ITU-T subjective tests the Pearson’s CorrelationCoefficient (R) was 0.935

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYNON-INTRUSIVE ASSESSMENT

A challenging problem since a reference is not available.Two approaches exist

1 Signal-based models2 Parametric models

Signal-based modelsRecent approaches are based on emulating

1 Human speech production model2 Psychoacoustic processing of human ear

ITU-T P.563 is the current Recommendation.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYNON-INTRUSIVE ASSESSMENT

A challenging problem since a reference is not available.Two approaches exist

1 Signal-based models2 Parametric models

Signal-based modelsRecent approaches are based on emulating

1 Human speech production model2 Psychoacoustic processing of human ear

ITU-T P.563 is the current Recommendation.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

OBJECTIVE ASSESSMENT OF SPEECH QUALITYPARAMETRIC MEASUREMENT OF VOIP QUALITY

Functions of transport layer metrics and other measurablequantities.Cogent metrics may be:

Packet Loss RateVariable delay – jitterEnd-to-end delay. . .

Aimed at Real-time and continuous evaluation of quality

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

VOICE OVER IP – VOIP

Packet based communication channelUses wire-line speech codecsLinear Predictive Coding (LPC) is having vogueCoded frames are packetized into RTP/UDPInternet is used for transportationThe receiver does the reverse process

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

RESEARCH GOAL

Derivation of a VoIP listening Quality estimation model as afunction of transport layer metrics.Genetic Programming based Symbolic Regression is usedUsing the PESQ algorithm as the reference system

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

VOIP SIMULATION ENVIRONMENTPACKET LOSS SIMULATION – THE GILBERT ELLIOT MODEL

mlr = pp+q (1)

mbl = 1q (2)

clp = 1− q (3)mbl = 1

1−clp (4)

Wheremlr – mean loss ratembl – mean burst lengthclp – conditional loss probability

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

VOIP SIMULATION ENVIRONMENT

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

NETWORK TRAFFIC PARAMETERS

No. Parameter Name Abbreviation1 Bit-rate (kbps) br2 mean loss rate mlr3 mean burst length mbl4 Packetization Interval (ms) PI5 Frame duration (ms) fd

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

NETWORK TRAFFIC SCENARIOS

No. Parameter Range1 br G.729 (8 kbps), G.723.1 (6.3 kbps),

AMR 7.4 and 12.2 kbps2 mlr [0,2.5,3.5,. . . 15,20,25,. . . 40]%3 mbl 10, 50, 60, 70 and 80%4 PI 10-60 ms5 fd 10, 20, 30 ms

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

EXPERIMENTAL SETUP

GPLabFour GP Experiments were performed with variousconfigurationsCommonalities

Each experiment constituted 50 runsEach Run spanned 50 generations

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

GP EXPERIMENTSCOMMON PARAMETERS

Parameter ValueInitial Population Size 300Selection LPP TournamentTournament Size 2Genetic Operators Crossover and Subtree MutationInitial Operator probabilities 0.5 initial, adaptive onwardsSurvival Half ElitismFunction Set +, -, *, /, sin, cos, log2, log10,

loge, sqrt, power,Terminal Set Random numbers [0.0 . . . 1.0]

Integers [2 . . . 10]. mlrVAD,mblVAD, PI, br , fd

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

GP EXPERIMENTSEXPERIMENTAL DETAILS

Experiment 1:Fitness function – Mean Squared Error MSE

Experiment 2:Linear Scaling MSEs

MSEs(y , t) = 1/nn∑i

(ti − (a + byi))2 (5)

a = t − by , b =cov(t , y)

var(y)(6)

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

GP EXPERIMENTSEXPERIMENTAL DETAILS

Experiment 1:Fitness function – Mean Squared Error MSE

Experiment 2:Linear Scaling MSEs

MSEs(y , t) = 1/nn∑i

(ti − (a + byi))2 (5)

a = t − by , b =cov(t , y)

var(y)(6)

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

GP EXPERIMENTSEXPERIMENTAL DETAILS

Experiments 3 and 4Selection criterion based on Gustafson et al. was usedMating takes place between dissimilar individuals

Experiment 4:The Maximum tree depth was reduced to 7 from 17

The results were treated to Mann-Whitney-Wilcoxon Testfor significance AnalysisExperiment 4 was found to be significantly better overall.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

GP EXPERIMENTSEXPERIMENTAL DETAILS

Experiments 3 and 4Selection criterion based on Gustafson et al. was usedMating takes place between dissimilar individuals

Experiment 4:The Maximum tree depth was reduced to 7 from 17

The results were treated to Mann-Whitney-Wilcoxon Testfor significance AnalysisExperiment 4 was found to be significantly better overall.

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

ON DATA COLLECTION

Nortel ND speech database containing high quality signalswith speech from 2 male and 2 female speakers was usedfor analysis.A total of 3360 distorted speech files were created for eachcombination of network traffic parameters.

1177 35% were used for training503 15% were used for testing1680 50% were used for speaker independent validation

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

VOIP QUALITY MONITORING MODELS

MOS − LQOGP = −2.46× log(cos(log(br)) + mlrVAD

×(br + fd/10)) + 3.17 (7)

MOS − LQOGP = −2.99× cos(0.91×√

sin(mlrVAD)

+mlrVAD + 8) + 4.20 (8)

Equation (7) Equation(8)Data MSEs σ MSEs σ

Training 0.0370 0.9634 0.0520 0.9481Testing 0.0387 0.9646 0.0541 0.9501Validation 0.0382 0.9688 0.0541 0.9531

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

SCATTER PLOTS

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

SCATTER PLOTSON PERFORMANCE OF ITU-T P.563

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

CONCLUSIONS

1 The model is a good approximation to PESQ.2 Suitable for real-time and non-intrusive estimation of

speech quality whereas PESQ is NOT.3 Simple models; depend on 3 and 1 variable respectively.4 Performs significantly better than ITU-T P.563

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

FUTURE GOALS

To include wide band codecs in the research.To develop a unified quality estimation model for narrowand wide band telephony

Motivation VoIP Simulation Environment GP Experiments Test Results Conclusions

QUESTIONS