CRA NETWORKAUDIT 2020QOS MEASUREMENTS SUMMARY
Communications Regulatory Authority
2
TABLE OF CONTENT
1. INTRODUCTION........................................................................................................................... 3
2. QUALITY OF SERVICE AUDIT.................................................................................................... 3
2.1 Measurement Methodology.............................................................................................................. 3
2.1.1 Voice Service Quality Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 Short Message Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.3 Data Service Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Equipment Used...................................................................................................................................... 4
2.3 Measurement Timeline........................................................................................................................ 4
2.4 Samples (Rounded)................................................................................................................................. 4
2.5 Voice Service.............................................................................................................................................. 4
2.5.1 Call Completion Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.5.2 Unsuccessful Call Attempt Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.5.3 Call Setup Time. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.5.4 Voice Quality (MOS). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.6 Short Message Service.......................................................................................................................... 7
2.7 Data Services to Operator’s Server................................................................................................. 8
2.7.1 Data Download Throughput – Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.7.2 Data Download Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.7.3 Data Upload Throughput – Maximum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.7.4 Data Upload Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.7.5 Webpage Download Success Rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.8 Data Services to International Servers....................................................................................... 10
2.8.1 Data Download Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.8.2 Data Upload Throughput – Distribution & Cumulative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
CRA Network Audit 2020 QoS Measurements
3
1. INTRODUCTION
As per the mandate provided to the Communications Regulatory Authority, to safeguard the interest of telecoms consumers, to publish information regarding the performance including comparisons of Quality of Service of the Service Providers, CRA has conducted the Quality of Service (QoS) Network Audit for the year 2020. The summary of the findings of the audit are as below.
2. QUALITY OF SERVICE AUDIT
The QoS audit is conducted to benchmark the Key performance Indicators of the services viz. Voice calls, Short message service (SMS) and Data service offered to the consumers. The samples are collected based on normal consumer behavior and from areas weighted based on the population density for the State of Qatar.
2.1 Measurement Methodology
2.1.1 Voice Service Quality TestingA voice measurement is a call to a Speech Quality Server for one minute duration and Mean Opinion Score (MOS) is recorded using POLQA algorithm.
2.1.2 Short Message ServiceA 52 character sample SMS is send automatically and the received timings are recorded. Tests are conducted within and across networks.
2.1.3 Data Service Testing• Data throughput measurement is carried out in a stationary environment by calculating
the average throughput for completion of an entire session from/to a server located within the operator’s network. Different file size and their respective maximum permissible timeouts as mentioned in the below table are used while collecting the samples.
Measurement Size / Timeout
Dowload (MB) 10 50 100
Upload (MB) 5 25 50
Timeouts (seconds) 150 300 300
• Webpage accessibility measurement is carried out in drive test and stationary environment which includes download a standard test page from the server.
Communications Regulatory Authority
4
2.2 Equipment Used• 4 UE’s - LTE CAT 16 device (Smartphone).• 2 UE’s - LTE CAT 18 device (Smartphone).• 2 UE’s - 5G enable device CAT 24 (Smartphone).• R&S Bench marker II Drive test software.• Handheld device - LTE CAT 16 device with Qualipoc software.
2.3 Measurement Timeline• 07th September 2020 till 31st December 2020.
2.4 Samples (Rounded)• Voice Service – 30900• Voice (Speech Quality) – 12700• Short Message Service – 5600• Data Service – 5800 • Web Service – 78800
KPI Definition
Call Completion Rate
Probability that a successful call attempt is maintained for 1 minute until it is released intentionally by user.
Number of normally ended callsCall completion rate = x100
All successful call attempts by all users
Unsuccessful Call Attempt Rate
The call set-up time is the time period between sending of complete address information and receipt of call set up notification.
Unsuccessful telephony service attempts by all users when service shown as available
Unsuccessful call attempt rate = x100All telephony service attempts by all users
Call Set Up Time The call set-up time is the time period between sending of complete address information and receipt of call set up notification.
Call setup time=t (Connect established) - t (User pressed button on terminal)
Voice Quality (MOS)
Voice quality in mobile networks is measured with algorithms based on ITU-T P.863 (POLQA), which covers the overall listening speech quality from narrowband (300 to 3’400 Hz) to super wideband (50 to 14’000 Hz) telecommunication scenarios as perceived by the user. The Mean Opinion Score (MOS) in both uplink and downlink, averaged for a call duration of 1 minute is reported.
2.5 Voice Service
CRA Network Audit 2020 QoS Measurements
5
Call Completion Rate
Statistical Accuracy ±0.03% ±0.02%
2.5.1 Call Completion Rate
2.5.2 Unsuccessful Call Attempt Rate
Unsuccessful Call Attempt Rate
Statistical Accuracy ±0.07% ±0.09%
50%
60%
70%
80%
90%
100%
99.91% 99.96%
0.00%
0.20%
0.40%
0.60%
0.80%
0.34%
0.60%
Communications Regulatory Authority
6
2.5.3 Call Setup Time
2.5.4 Voice Quality (MOS)
Average Call Setup Time (Sec)
Average Voice Quality (MOS)
0.00
1.00
2.00
3.00
4.00
3.81
3.20
0.00
1.00
2.00
3.00
4.00
5.00 3.82 3.79
CRA Network Audit 2020 QoS Measurements
7
SMS Successful Completion RateSMS Successful Completion Rate(Own Network)(Own Network)
Statistical Accuracy ±0.23% ±0.14%
KPI Definition
SMS Successful Completion Rate (Own & Cross Network)
Number of SMS transfer completedSMS success rate = x100
Number of SMS sent successfully
SMS End to End Transfer Time
The SMS end to end delivery time is the period of time between sending a short message to the network and the message being received at the distant terminal (user device)
SMS end to end delivery time= t (B,received) - t (A,send initiate)
2.6 Short Message Service
SMS Successful Completion RateSMS Successful Completion Rate(Cross Network*)(Cross Network*)
Statistical Accuracy ±0.23% ±0.27%
SMS Avg End to End Transfer Time (Sec)SMS Avg End to End Transfer Time (Sec)(Cross Network*)(Cross Network*)
SMS Avg End to End Transfer Time (Sec)SMS Avg End to End Transfer Time (Sec)(Own Network)(Own Network)
*Note: The originating operator’s does not have absolute control on Cross network KPI’s which are terminated in the other network.
95%
96%
97%
98%
99%
100%
99.61% 99.86%
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1.56
2.46
0.0
0.5
1.0
1.5
2.0
2.5
3.02.46
1.64
95%
96%
97%
98%
99%
100%99.61% 99.44%
Communications Regulatory Authority
8
KPI Definition
Data Download Throughput
Downloading a file via HTTP.
The Average throughput to download the entire session is calculated.
2.7 Data Services to Operator’s Server
Average Download Throughput - Maximum
Average Download Throughput - Local Server
2.7.1 Data Download Throughput – Maximum
2.7.2 Data Download Throughput – Distribution & Cumulative
CDFCDF
0
100
200
300
400
500
600
700
800
743.89 Mbps
600.79 Mbps
1.8% 2.8%4.4%
6.3% 8.2% 10.6%15.4%
22.8%
39.8%
54.6%73.6%
90.0%96.4%
100.0%
0.8% 2.4%6.5% 11.4%
16.9% 21.8%28.0% 38.6%
53.7%62.3%
76.2%
89.7%96.4% 100.0%
0%10%20%30%40%50%60%70%80%90%100%
0
50
100
150
200
250
300
>500 450 400 350 300 250 200 150 100 75 50 25 10 <10
# S
ampl
es
Mbps
CRA Network Audit 2020 QoS Measurements
9
Average Upload Throughput - Maximum
KPI Definition
Data Upload Throughput
Uploading a file via HTTP.
The Average throughput to upload the entire session is calculated.
Average Upload Throughput - Local Server
2.7.3 Data Upload Throughput – Maximum
2.7.4 Data Upload Throughput – Distribution & Cumulative
CDFCDF
0.7% 2.9%11.1%
30.5%
59.3%
83.1%94.3%
100.0%
0.6% 5.1%14.5%
30.1%
46.4%
69.5%
89.1%100.0%
0%10%20%30%40%50%60%70%80%90%100%
050
100150
200250300350400450
>70 60 50 40 30 20 10 <10
# S
ampl
es
Mbps
010
20304050607080
78.46 Mbps 75.42 Mbps
Communications Regulatory Authority
10
Successful Webpage Download Rate
Statistical Accuracy ±0.01% ±0.02%
2.7.5 Webpage Download Success Rate
KPI Definition
Successful Webpage Download Rate
Number of completed download sessionSuccessful webpage download rate = x100
Number of session started successfully
CRA conducted test to multiple international servers located in different locations outside Qatar, to portray typical user experience for data sessions. The average throughputs across different test samples are summarized below. The results are indicative and represents throughputs to the International servers, chosen by CRA and the identity of the locations of the servers are kept anonymous to the service providers. The throughput rates can vary depending on various factors for e.g. Internet traffic, Location of the server which may be beyond the operator’s absolute control.
2.8 Data Services to International Servers
50%
60%
70%
80%
90%
100%
99.98% 99.93%
CRA Network Audit 2020 QoS Measurements
11
2.8.1 Data Download Throughput – Distribution & Cumulative
2.8.2 Data Upload Throughput – Distribution & Cumulative
Average Download Throughput - International Server
Average Upload Throughput - International Server
CDFCDF
CDFCDF
0.0% 3.8% 6.0% 9.6%14.7%
21.3%27.8%
39.2%
56.4%
79.6%92.3%
100.0%
10.2%18.3% 19.3% 20.1%
24.4%35.3% 45.8%
53.4%
67.9%
85.4%95.1% 100.0%
0%10%20%30%40%50%60%70%80%90%100%
050
100150
200250300350
>150 100 90 80 70 60 50 40 30 20 10 <10
# S
ampl
es
Mbps
0.6% 1.3%5.6%
19.2%
43.4%
86.6%100.0%
0.8% 3.8%
10.0%21.4% 41.7%
67.1%
100.0%
0%10%20%30%40%50%60%70%80%90%100%
0
200
400
600
800
>60 50 40 30 20 10 <10
# S
ampl
es
Mbps
Communications Regulatory Authority
12www.cra.gov.qa
Top Related