Wp Testing Wireless Vq
-
Upload
yousufalqeisi -
Category
Documents
-
view
220 -
download
0
Transcript of Wp Testing Wireless Vq
-
8/2/2019 Wp Testing Wireless Vq
1/16
| W H I T E P A P E R |
Testing Wireless Voice Quality:
Capturing the Customer Experience
December 2006
-
8/2/2019 Wp Testing Wireless Vq
2/16
Contents
Copyright 2007 Ditech Networks 2/16
Executive Summary.......................................................................................................................... 3
Voice Quality The Customer Experi ence.......................................................................................... 4
Increased Network Complexity...................................................................................................... 4
Rising Customer Expectations....................................................................................................... 5
Put Your Company Ahead o the Curve with Di tech Networks......................................................... 6
VQA....................................................................................................................................6
Experience Intelligenc (EXi)....................................................................................................6
Overview o Voice Quality Testing..................................................................................................... 7
Standards Governing Voice Quality Testing.................................................................................... 7
ITU P.800............................................................................................................................... 7
ITU P.862............................................................................................................................... 8
ITU G.107.............................................................................................................................. 9
Current Industry Tools or Asse ssing Voice Quality.........................................................................10
Intrusive Tests or Characterizing the Network..............................................................................10
Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience....................... 11
Exposing External Impairments....................................................................................................11
Noise and Speech Level......................... ................................................................................. 12
Echo and Delay......................................................................................................................12
Delivering Meaningul Customer Experience Data......................................................................... 12A Future-Proo Solution..........................................................................................................14
Value or Multiple Quality Initiatives........................................................................................14
Control the Voice Quality Exper ience............................................................................................... 15
Testing Wireless Voice Quality
-
8/2/2019 Wp Testing Wireless Vq
3/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks /16
Executive Summary
A customers experience o voice quality extends beyond the physical network, as environmental issues and user
equipment can have a proound impact on how a call sounds. The users environment and equipment impact the
voice quality or a subscriber even in a perect RF environment because they directly aect the speech signal input.
Recognizing this issue is important because customers are becoming less tolerant o voice problems as they become
more sophisticated wireless users.
Existing industry voice quality tests ocus on the network, primarily radio requency (RF), and not on the speech signal.
The ollowing quotation rom the documentation o a leading wireless voice quality test tool illustrates this bias:
The Speech Quality Index (SQI) is a patented Ericsson algorithm which is built upon the act that
almost all speech distortions in a mobile network are due to problems with the radio transmission.1
RF- and codec-ocused tests are not unimportant, but they provide a severely limited assessment o the overall
customer experience. In addition to ocusing only on network component perormance, existing voice quality tests
are intrusive, requiring the injection o sampled speech signals and measurement o the post-processed signal. This
methodology provides no insight into the number o customer calls impaired by the users environment or sub-par
equipment.
Ditech Networks Experience Intelligence (EXi) experience intelligence solution provides the industrys frst and only
live monitoring o environmental anduser-equipment-related impairments on every call, all the time. Ditechs EXi
solution uses the ITU-Ts G.107 standard to provide Transmission Rating (R) Factor and Mean Opinion Score (MOS)
values on the combined measured impact o background noise, echo, and audio level impairments on every call.
EXi provides an unprecedented visibility into the customers real-lie call experience.
1. Speech Quality Index in CDMA2000, published by Ericsson.
-
8/2/2019 Wp Testing Wireless Vq
4/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 4/16
Voice Quality The Customer ExperienceThe ollowing trends are impacting the customers voice quality experience:
1. Network complexity is introducing additional voice quality impairments.
2. Customer expectations o voice quality rom wireless phones are rising, so mere annoyances in years past
are now creating churn.
3. As mobile become more commonplace, more calls originate rom andend in noisy or acoustic-echo prone
situations, adding voice quality issues.
Figure 1 outlines the issues involved in managing the customers experience o voice quality:
Figure 1 : : Impairments Experienced by the Customer
Increased Network Complexity
The world o voice communications is becoming increasingly complex. Despite industry consolidation, device and
IP-telephony prolieration are making it more difcult or carriers to engineer high voice quality. These trends intro-
duce impairments beyond the physical boundaries o the network, but directly impacting the customer experience
o the network.
Carriers test handsets beore they are oered to customers. In the early days, carriers could exhaustively test the
small number o handsets in their oerings. But today, the number o devices on the market is increasing and device
release cycles are shortening, making it difcult or carriers to dedicate the resources to extensively test each new
model. Testing is urther complicated by the intense competitive pressures to quickly introduce new models and tailor
handset oerings to multiple market segments. The inability to tightly control voice quality in a wide range o hand-
sets rom multiple vendors has resulted in greater incidences o acoustic echo, inappropriate audio levels, and noise
introduction.
Phone
Environment Other Networks
SIP Phone
Ability to Manage and Mitigate
Phones
Network
Accessories
LOW
LIMITED
COMPREHENSIVE
-
8/2/2019 Wp Testing Wireless Vq
5/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 5/16
I controlling voice quality among multiple handset oerings is difcult, controlling quality once atermarket accessories
are added to a handset is impossible. The market is awash with device accessories, which consumers purchase o the
shel and oten personalize. The millions o possible device interaction permutations cannot be tested economically.
One aspect o a call that a carrier can control is its own network, and the industry has developed a robust methodology
or network characterization. In network testing, two attributes are primary used to defne voice quality:
Radio requency (RF) Assesses the wireless link perormance using metrics such as rame erasure rate (FER), bit
error rate (BER), and carrier-to-intererence ratio (C/I).
Network-wide quality Assesses network delay and codec selection.
However, this network control ends at the network edge, where trafc is delivered to other carriers or routing within
their networks. Unortunately, these interconnections oten introduce acoustic and hybrid echoes, noise, varying audio
levels, and delay, which add to poor voice quality.
Rising Customer Expectations
In addition to the mounting complexity in simply maintaining the status quo in voice quality, there is evidence rom
a leading research survey that consumer expectations o voice quality are increasing as the wireless industry matures.
This has resulted in a marked reduction in tolerance or voice quality issues such as noise, echo, and level mismatch.
The survey went urther to identiy that this reduction in tolerance is directly related to churn. In eect, service quality
issues that did not cause a customer to consider switching services last year are now considered unacceptable and
cause churn.
Figure 2 shows the voice quality impairments discussed above and where they originate within the overall call path.
Figure 2 : : Voice Quality Impairment Origins in the Network
SIP Phone
Acoustic EchoNoise
Audio LevelHybrid Echo
CodecDelay
Acoustic EchoAmbient Noise
Acoustic EchoAudio Level
FERBERC/I
Environment Device RF Network Wide Other Networks
BSC MSC
PSTN
-
8/2/2019 Wp Testing Wireless Vq
6/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 6/16
Put Your Company Ahead of the Curve with Ditech NetworksWhat can be done to mitigate the problems that come rom outside a network, but directly impact customer experi-
ence and cause churn?
With the ollowing products, Ditech Networks oers the most comprehensive solution or managing and improving
the customer experience beyond the reach o a wireless carriers network boundaries:
VQA The most comprehensive suite of voice improvement capabilities
Superior Voice Quality Improvement
- Advanced Noise Cancellation (ANC)
Cancels noise without distorting the speakers voice
Eective even in extremely low SNR environments
- Automatic Level Control (ALC)
Removes level mismatches caused by both the network and the device
Prevents clipped speech and distortion
- Enhanced Voice Intelligibility (EVI)
Improves speech clarity, especially or calls that use low bit-rate codecs in noisy environments
Speech-based signal processing without ampliication or distortion
Dramatic statistical word recognition improvement
Advanced Echo Control
- Industry-best Hybrid Echo Cancellation (HEC)
192 ms tail length, the highest in industry
- Bidirectional Acoustic Echo Control (AEC)
Cancels echo a ull 3 dB louder than the competition
DSP-enabled nonlinear processing to address unique acoustic echo challenges
EXi The industrys only real-time customer experience metrics that include external impairments
24/7 measurement o every call
Provides visibility to impairment issues outside o the carriers network
Measures noise, echo, delay, and audio level impairments
Delivers ITU-T G.107-based R Factor and MOS values or every call
-
8/2/2019 Wp Testing Wireless Vq
7/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 7/16
Overview of Voice Quality TestingHow do wireless carriers test or voice quality impairments and remedy them? This section discusses the capabilities and
limitations o existing standards and the industry tests associated with voice quality testing.
Standards Governing Voice Quality Testing
The most common standards that assess the impact o various elements on voice quality are the International Telecom-
munication Unions Telecommunication Standardization Sector (ITU-T) Recommendations P.800, P.862, and G.107. These
standards were designed to address specifc voice quality issues, as outlined in Table 1.
Table 1 : :Focus o ITU-T Standards or Voice Quality Testing
Standard Summary Environment Device RF Network Other Network
ITU-T P.800 Subjective Voice Quality Test Level All Codec Noise LevelITU-T P.862 Objective Voice Quality Test PESQ All Codec
ITU-T G.107 E-Model or Voice Quality Analysis All All All All
ITU-T P.800
ITU-T Recommendation P.800 identifes a subjective standard that attempts to measure listening quality and listening eort
using Absolute Category Rating (ACR), Degradation Category Rating (DCR), and Comparison Category Rating (CCR) pro-
cedures. These procedures require a panel o listeners to assign voice calls a score based on specifc criteria. These scores
are then aggregated into Mean Opinion Scores (MOS) that are quantifed rom 1 (worst) to 5 (best).
ITU-T P.800 is limited in its ability to provide accurate results when assessing the perormance o a voice enhancement
device. Dynastat, a leader in the subjective testing industry stated the ollowing in an ITU-T submission:
Standard rating scale methodologies, e.g., ACR, DCR, and CCR, can give conounding and even
spurious results when used to evaluate the eects o [voice enhancement devices].2
The paper went on to postulate that the inability or people to reliably rate quality is that individuals cannot weigh the
value o the two variables in voice enhancement o modifcation to the original signal and noise reduction level in a
repeatable manner.
Thereore, ITU-T P.800 is o limited use in assessing the impact o noise in the network since it cannot be relied upon to
provide reliable results on the change in customer experience created by introducing noise reduction technology. ITU-T
P.800 is also impractical due to the high cost o perorming subjective testing or network monitoring purposes.
2. From ITU-T Submission AH-00.
-
8/2/2019 Wp Testing Wireless Vq
8/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks /16
ITU-T P.862
ITU-T P.862s Perceptual Evaluation o Speech Quality (PESQ) procedure is designed to be analogous to the ITU-T P.800
procedure, providing an objective method or predicting a subscribers experience o voice quality rom a signal that
passes through a communication system. PESQ uses a defned series o phrases to produce a score that can be directly
mapped to the MOS value. However, PESQ is only accurate or a limited range o circumstances, none o which impact
the speech signal itsel rom noise, echo, or level issues. Additionally, PESQ only measures during active speech no
weight is given to the listening experience in pauses or silence.
The three tables below are rom the ITU-T P.862 Recommendation and identiy the capabilities and limitations o PESQ.
PESQ has demonstrated acceptable accuracy or the applications listed in Table 2.
Table 2 : : Factors or Which PESQ Has Acceptable Accuracy (ITU-T P.862)
Codec evaluation
Codec selection
Live network testing using digital or analog connection to the network
Testing o emulated and prototype networks
Note: When environmental noise is present, the quality can be measured by passing PESQ the clean original without noise, and the degraded
signal with noise.
PESQ is known to provide inaccurate predictions when used in conjunction with the variables in Table 3, or is otherwise
not intended to be used with these variables.
Table 3 : : Factors or Which PESQ Has Inaccuracy (ITU-T P.862)
TestFactorsListening levels (see note below)
Loudness loss
Eect o delay in conversational tests
Talker echo
Sidetone
Coding Technologies
Replacement o continuous sections o speech making up more than 25% o active speech by silence
(extreme temporal clipping)
Applications
In-service non-intrusive measurements device
Two-way communications perormance
Note: PESQ assumes a standard listening level o 79 dB SPL and compensates or non-optimum signal levels in the input fles. The subjective
eect o deviation rom optimum listening level is thereore not taken into account.
-
8/2/2019 Wp Testing Wireless Vq
9/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks /16
PESQ has not been validated or measuring the test actors and applications listed in Table 4.Table 4 : :Factors or Which PESQ Is Not a Valid Test (ITU-T P.862)
Test Factors
Music as input to a codec
Listener echo
Eects/artiacts rom operation o echo cancellers
Eects/artiacts rom noise reduction algorithms
Coding technologies
CELP and hybrid codecs
-
8/2/2019 Wp Testing Wireless Vq
10/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 10/16
Current Industry Tools for Assessing Voice QualityExisting testing tools are based primarily on the P.862 PESQ standard and on the Spirent (ormerly SwissQual) SQUAD-
LQ proprietary algorithm. These tools include intrusive tests, using recorded fles, to measure voice quality within the
deterministic network environment. These tests have two primary limitations or measuring aspects o the voice quality
customer experience outside o the network component:
1. The tests rely on speech iles to characterize the network perormance rather than capture actual call metrics.
2. The algorithms have a limited ability to accurately assess non-network-based impairments to customer experience
even or recorded speech iles due to the limitations o the PESQ and SQUAD tools in modeling speech signal
impairments (as was identiied in the previous discussion o ITU P.862).
Table 5 below identifes the capabilities o various toolsets with respect to the overall customer experience.
Table 5 : : Ability o Various Tools to Measure Overall Customer Experience
Tool Algorithm Live Calls Environment Device RF Network Other Network
TEMs PESQ No All Codec
TEMs SQI Yes All Codec
SwissQual SQUAD-LQ No Audio Level All Codec
Invex3G PESQ No All Codec
Invex3G SQUAD-LQ No Audio Level All Codec
QVoice PESQ No All Codec
Agilent PESQ No All Codec
GL PESQ No All Codec
Intrusive Tests for Characterizing the Network
The objective o intrusive tests is to characterize the network while minimizing introduced impairments. Intrusive testing
is a very eective tool or this because the network component o the voice quality experience is a controlled environ-
ment that will provide consistent output or consistent input. Thereore, under a given network load and with a defned
RF environment the network will deliver the same quality level every time.
The same is not true or impacts to the speech signal that occurs on actual customer calls. A noisy background can cause
degradation to the customer experience even in a high-quality RF environment. The same is true o acoustic echo or im-
proper speech level coming in rom another network or a poor device. These issues cannot be identifed using recorded
fles, but only through an assessment o actual calls.
-
8/2/2019 Wp Testing Wireless Vq
11/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 11/16
SIP Phone
Acoustic EchoNoise
Audio LevelHybrid Echo
CodecDelay
Acoustic EchoAmbient Noise
Acoustic EchoAudio Level
FERBERC/I
Environment Device RF SNRINPLR
WAEPLEPL
MOSR-Value
VED
Network Wide Other Networks
BSC
MSC
PSTN
Ditech Networks EXi Solution: Managing the Customers Live Voice Quality Experience
Ditechs EXi solution delivers new visibility into impairments that occur during real calls, accurately identiying where and
when impairments occur, and quantiying the impairments impact on the subscribers experience.
Ditechs EXi provides the ollowing eatures:
Measurement o live customer experience
Visibility to impairments outside o the carriers network
R Factor and MOS value per call based on measured noise, echo, and level impairments
Table 6 l ists EXis current and uture capabilities and eatures.
Table 6 : :Ditech Networks EXi Capabilities
Tool Algorithm Live Calls Environment Device RF VED Network Other Network
Ditech EXi ITU-T G.107 Yes All All Future All Future All
Accurate Production Environment Measurements
Ditechs EXi provides 24-hour monitoring on a per-call basis. Live monitoring delivers accurate measurements o subscrib-
ers actual experiences using a production network.
Exposing External Impairments
With Ditechs EXi technology, or the frst time carriers have visibility into impairments that originate outside o their
network. Ditechs solution monitors or acoustic echo, noise, audio level, and hybrid echo pinpointing the impairments
and removing them. Ditechs EXi also helps the carrier identiy problems caused by other networks, urther aiding in
troubleshooting.
EXi measures the ollowing voice quality impairments:
Noise level
Speech level
Hybrid echo delay
Hybrid echo return loss
Acoustic echo delay
-
8/2/2019 Wp Testing Wireless Vq
12/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 12/16
Noise and Speech Level
An increasing number o calls originate rom noisy urban environments, and background noise can be a major degrada-
tion actor to the listening quality o a call. Ditechs EXi measures average per-call speech level, noise level, and signal-to-
noise ratio (SNR) in both directions on every call.
Echo and Delay
The migration o voice networks rom circuit to packet technologies is resulting in greater and more variable latency. The
majority o legacy hybrid echo cancellers are unable to handle this added delay, leaving callers unprotected rom annoying
hybrid echo.
In addition, variability in terminal equipment (handsets, headsets, and handsree kits) continues to expand without
necessarily providing adequate acoustic isolation, exposing more users to acoustic echo.
Ditechs EXi provides comprehensive statistics about echo delay and return loss or both hybrid (linear) echo and acoustic
(non-linear) echo. It monitors rom 0 to 400 ms in both directions, ensuring complete and accurate measurement coverage.
For a statistical understanding o how echo and delay combine to aect customer satisaction, Ditechs EXi also provides
Echo Objection Rates per ITU-T G.131.
Delivering Meaningful Customer Experience Data
Ditechs EXi solution delivers a unifed MOS score o the actual subscriber voice quality experience, measuring all calls, all
the time, or multiple quality impairments. The solution also complements existing test equipment or easy deployment
into an existing carrier environment.
Per-call listening and conversational R Factors and MOS values are continuously and non-intrusively computed based on
the measured voice quality impairments (speech, noise, and echo), as well as codec type. Ditechs objective scores ollow
the ITU-T G.107 E-Model standard using an industry-unique approach o DSP-based measurements o the live voice signal.
Ditech has partnered with Telchemy, a leading provider o perormance monitoring technology, and employs their
VQmon computational model to provide R Factors and MOS values or the ollowing:
Listening Quality (LQ) This score includes noise and speech level. It does not include impairments that aect
conversation, such as delay.
Conversational Quality (CQ) This score includes the impairments measure or the LQ score and adds echo and delay,
which aect conversational quality.
-
8/2/2019 Wp Testing Wireless Vq
13/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 1/16
The impact o the impairments is reported in any o the ollowing mechanisms:
Transmission Rating (R) Factors
o Listening Quality (R-LQ)
o Conversational Quality (R-CQ)
Mean Opinion Scores (MOS)
o Listening Quality (MOS-LQ)
o Conversational Quality (MOS-CQ)
Weighted acoustic echo path loss
Echo Objection Rate per ITU-T G.131
Ditech EXi data can be aggregated rom the local, regional, and national levels to gain visibility into the customer experi-
ence at any point in the network. Results are easily integrated into existing management and executive dashboards or
at-a-glance status reporting. Below is an example o how EXi data can be graphically presented to show the level o voice
quality customers are experiencing.
Figure 3 : : Sample EXi Data
MOS Conversational Quality
1 2 3 4 5
18,000
16,000
14,000
12,000
10,000
8,000
6,000
4,000
2,000
MOS Score
CallCount
2,4471,254
14,750 15,207
8,923
14,563
12,328
14,332
10,432
16,100
Rin
Rout
-
8/2/2019 Wp Testing Wireless Vq
14/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 14/16
A Future-Proof Solution
Ditech EXi uses a Telchemy scoring algorithm that delivers an R Factor and MOS values per call. The Telchemy algorithm is
widely adopted in IP networks, with more than two million licenses issued. This ensures that as wireless networks evolve
to VoIP, the quality measurements initiatives put in place today will continue to be o value in the uture.
Value for Multiple Quality Initiatives
Ditech EXi is easily integrated with other tools rom multiple vendors to improve voice quality management and expand
visibility into call quality and the origin o impairments. MOS data can be correlated with other network data, such as cell
identifer and Electronic Serial Number (ESN).
Integrating these multiple data points provides wireless carriers the ability to recognize and identiy network trouble
points, receive automatic alerts when a problem arises, and provide advanced reporting or network planning and
customer Service Level Agreements (SLAs).
Figure 4 : : Managing Network-wide Voice Quality
Partners Ditech
-
8/2/2019 Wp Testing Wireless Vq
15/16
Testing Wireless Voice Quality
Copyright 2007 Ditech Networks 15/16
Control the Voice Quality ExperienceDitech EXi provides the ability to manage the complete end-to-end voice quality experience, so that wireless carriers can
use their existing PESQ, SQUAD-LQ, or SQI tools or RF-related network quality measurements, and EXi or monitoring
their noise, echo, and level impairments (Figure 5).
Figure 5 : : Tools or Controlling the Customer Experience
Ditechs EXi and VQA solutions provide carriers with visibility into andmitigation o impairments that, until now, have
been outside o a carriers control. Used in conjunction with industry-standard voice quality network tests, wireless carriers
now have tools to identiy and mitigate all aspects o voice quality impairments impacting the customer experience.
Phone
Environment Other Networks
SIP Phone
Phones
Network
Accessories
Poor Fair Good
Poor Fair Good
Good Good Fair
PESQ SQUAD-LQ Ditech EXi
-
8/2/2019 Wp Testing Wireless Vq
16/16
Testing Wireless Voice Quality
Corporate Headquarters
US Sales West
Ditech Networks
825 East Middleield Rd.
Mountain View, CA 94043
USA
1-800-234-0884 (Toll Free)
1-650-623-1300 (Direct)
1-650-564-9599 (Fax)
US Sales Office East and Central
Ditech Networks
8360 Greensboro Dr.
McLean, VA 22102
USA
1-800-234-0884 (Toll Free)
1-650-623-1300 (Direct)
1-650-564-9599 (Fax)
Canada Sales Office
Ditech Networks
2275 Lakeshore Blvd West, Suite 500
Toronto, ON M8V 3Y3
Canada
1-416-255-7776 (Phone)
Mexico Sales Office
Ditech Networks
Torcuato Tasso 245-6o. Piso
Col. Polanco
Mxico, D.F. C.P. 11570
Mxico
+52 (55) 5254-5422 (Main)
+52 (55) 5350-8679 (Sales)[email protected]
Brazil Sales Office
Ditech Networks
Alameda Araguaia, 933
Alphaville
Barueri, SP 06455-000
Brazil
+55 (11) 4208-6266 (Main)
+55 (21) 3521-5543 (Sales)
China Sales Office
Ditech Networks
Unit 3010, No. 500
Xiangyang South Road
Xuhui District
Shanghai 200031
PRC
+86 (21) 5456 0305 (Phone)
South-East Asia Sales Office
Ditech Networks
Lippo Plaza 3rd Floor
Jl. Jend. Sudirman Kav. 25
Jakarta 12920
Indonesia
+62 (21) 5291 3780 (Phone)
+62 (21) 522 1977 (Fax)
South Asia Sales Office
Ditech Networks
No. 8/11, Sarvapriya Vihar
New Delhi 110016
India+91 9810372555 (Phone)
Middle East/Africa Sales Office
Ditech Networks
21 El-Fawakeh St., 3rd Floor
Dokki, Giza 12311
Egypt
+20 2 336-5100 (Phone)
+20 2 761-8964 (Fax)
Spain Sales Office
Ditech Networks
Torres Quevedo, 1 (P.T.M.)
28760 Tres Cantos Madrid
Spain
+34 (91) 803 74 44 (Main)
+34 (91) 829 26 90 (Sales)
About Ditech NetworksDitech Networks supplies voice processing equipment or telecommunication networks around the world. Ditech Networks technology
solutions include voice, media processing, SIP, and security delivered on carrier-grade scalable platorms to enhance the delivery o
communications services over mobile, Voice over IP, and wireline networks. Ditech Networks customers are premier network operators
including Verizon, AT&T, Orascom Telecom, and others that collectively serve more than 150 million subscribers.