0 | P a g e

Independent Communications Authority of South Africa Pinmill Farm, 164 Katherine Street, Sandton

Private Bag X10002, Sandton, 2146

Quality of Service Report:

Limpopo Province

2016/2017 Quarter 1

1 | P a g e

Contents

EXECUTIVE SUMMARY.................................................................................................................................................... 3

1 Introduction ............................................................................................................................................................ 4

2 Methodology .......................................................................................................................................................... 5

2.1 Drive Test Equipment ......................................................................................................................................... 5

2.2 Route Selection .................................................................................................................................................. 5

2.3 Equipment test setup and configuration ........................................................................................................ 6

2.4 Statistical Significance .................................................................................................................................... 8

2.5 Measurement Parameters and Targets .......................................................................................................... 8

2.5.1 Parameter Targets ...................................................................................................................................... 8

2.5.2 Drop-Call Ratio ........................................................................................................................................... 8

2.5.3 The Call Setup Success Ratio ...................................................................................................................... 9

3 Results and Analysis ............................................................................................................................................. 10

3.1 Accessibility and Retainability Measurements ............................................................................................. 10

3.1.1 Call Setup Success Ratio (CSSR) ................................................................................................................ 11

3.1.2 Drop Call Ratio (DCR) ................................................................................................................................ 12

3.2 Technology Distribution ............................................................................................................................... 13

4 Conclusion ............................................................................................................................................................ 13

5 Appendixes ........................................................................................................................................................... 15

5.1 Appendix A: Drive Test Results KPI’s ................................................................................................................ 15

5.2 Appendix B: Coverage Maps ......................................................................................................................... 17

5.2.1 Accessibility: Signal Levels for Serving Cells ............................................................................................. 17

5.2.2 Retainability: Signal Levels for Serving Cells ............................................................................................. 18

5.2.3 Serving Technology Details ....................................................................................................................... 19

2 | P a g e

List of Acronyms

ACSSR Average Call Setup Success Ratio

ADCR Average Drop Call Ratio

CD Call Duration

CSSR Call Setup Success Ration

DCR Drop Call Ratio

GSM Global System for Mobile Communications

IVR Interactive Voice Response

WCDMA Wideband Code Division Multiple Access

MOC Mobile Originating Call

3 | P a g e

EXECUTIVE SUMMARY

The Authority conducted Quality of Service (QoS) measurements on the networks of

mobile operators; Cell C, MTN, Telkom, and Vodacom. The measurements were

performed to monitor performance of voice services being offered by the operators in

the Limpopo province. The measurements were carried out in April 2016 by drive

testing and covered a total distance of 3359 kilometres.

The purpose of collecting QoS measurements was to monitor and analyse the service

quality experienced by the end user of voice services offered. The measurements

were conducted in areas and in circumstances where mobile service usage is likely to

be accessed. These areas includes major towns, townships, farm areas, rural areas,

major road arteries, economic activity nodes and areas of previous complaints. The

sampled areas includes Ga-Riba/Ga-Nkoana, Tzaneen, Hoedspruit, Modjadjiskloof and

Bela-Bela.

A vehicle equipped with TEM Symphony measurement tool was used to collect data.

The two Key Performance Indicators (KPIs) measured were Retainability and

Accessibility. The Drop Call Rate (DCR) parameter is used to measure Retainability

and Call Setup Success Rate (CSSR) parameter measures Accessibility.

According to the End-User and Subscriber Service Charter regulations of 2016, DCR

should be less than 3% and CSSR should be greater than 98%.

In terms of the average results of the areas covered, all the operators CSSR was less

than 98% and therefore did not comply with the Accessibility target. Cell C, MTN and

Vodacom complied with the average Retainability target as their DCR was less than

3% and Telkom did not comply with the average Retainability target. Cell C performed

least on CSSR and Telkom performed least on DCR.

4 | P a g e

1 Introduction

ICASA’s mission is to ensure that all South Africans have access to a wide range of

high quality communication services at affordable prices1. The Authority ensures the

quality of service through its QoS monitoring activities. The Authority conducted

Quality of Service (QoS) monitoring of the GSM voice telephony service offered by

Cell-C, MTN, Vodacom and Telkom within Limpopo Province.

Limpopo is South Africa’s northernmost province, borders onto Mozambique,

Zimbabwe and Botswana. It also borders the Mpumalanga, Gauteng and North West

provinces. It ranks fifth in South Africa in both surface area and population, covering

an area of 125 754km² and being home to a population of 5 404 868. The capital is

Polokwane (previously known as Pietersburg). Other major cities and towns include

Bela-Bela (Warmbad), Lephalale (Ellisras), Makhado (Louis Trichardt), Musina

(Mesina), Thabazimbi and Tzaneen2. The province is divided into five district

municipalities, which are further subdivided into 25 local municipalities and it has the

largest percentage of households who had only cellular phones in their dwellings.3

Figure 1: Map of Limpopo Province with drive test routes

1 ICASA Strategic Plan 2016/17-2021 2 http://www.localgovernment.co.za/provinces/view/5/limpopo 3 StatSA (2015), General household survey 2014, P0138

5 | P a g e

QoS is defined as a collective effect of service performance that determines the degree

of satisfaction of a user of the service. QoS provides an indication of what the

customer experiences when using the mobile network and is evaluated in terms of

Retainability and Accessibility. Retainability is defined as the ability for a call to stay

connected through to a normal call tear-down process, without abnormally

disconnecting from the cell site that caries the call4. Accessibility is defined as the

percentage of the number of times a user is rejected due to the unavailability of

system resources when attempting to place a call.

The KPI parameters used to measure Retainability and Accessibility are Drop Call Rate

(DCR) and Call Setup Success Rate (CSSR), respectively.

The methodology used is the drive test sampling, which provides a snapshot view of

the mobile operator’s quality of service and provides a realistic picture of network

performance from a user’s point of view. It must be noted that drive test results do

not represent the mobile service provider's overall network performance. It is based

on the specified routes, at a particular time of day when the measurements were

carried out using a compliant handset. All the test phones were configured to

automatically select any network and technology.

2 Methodology

Drive testing sampling methodology was used in the assessment of the operators’

networks. This method provided an external indicative snapshot of an operator’s

network performance from the user’s point of view on the selected routes.

The drive test results represent a snapshot of the mobile service provider's network

performance, based on the specified routes during the time of day when the

measurements were conducted.

2.1 Drive Test Equipment

Drive test measurements was carried out using a test kit comprising of TEMS

Symphony 7.7 software, Samsung Galaxy Note 4 (SM-N910F) mobile device, a laptop

configured with Probe Controller v1.9.2.

2.2 Route Selection

Measurement routes were selected to reflect end-user distribution and different

geographical locations along areas where people live and use mobile phones (e.g.

4 Annexure B.2 of ETSI EG 202 057-3

6 | P a g e

urban, suburban, major towns, rural towns, township, farm areas, highways, tourism

areas and major roads).

Five areas were selected within the following district municipalities; Mopani,

Sekhukhune and Waterberg as indicated on the table below:

Table 1: Selected Routes and Date

Test Type District/Metropolitan

Municipality Route Name Dates

Accessibility

and

Retainability

Sekhukhune Ga-Riba/Ga-Nkoana

14/04/2016

15/04/2016

Mopani

Hoedspruit

17/04/2016

18/04/2016

Tzaneen

16/04/2016

21/04/2016

Modjadjiskloof

19/04/2016

20/04/2016

Waterberg Bela-Bela

22/04/2016

25/04/2016

2.3 Equipment test setup and configuration

The technical and methodological procedure followed used has a direct influence on

the results, therefore a concise test plan is important as defined in Table 2 below:

Table 2: Test plan and configurations

Configuration Explanation

Antennas

Tests were carried out from a moving vehicle with roof

mounted antennas. The antennas were arranged in a well-

defined fixed way with a minimum distance to each other,

reducing RF-influence to an acceptable level in an equal radio

7 | P a g e

environment. The coupling loss between two mobile

handsets was min 40.5 dB5.

Band The bands tested were GSM (900 and 1800) and WCDMA

(900, 2100)

Call Samples

A minimum of 120 test samples per network operator were

collected, except in the areas where services were limited on

most part of the drive test route. Test drives were planned

to ensure that the results adequately reflect the QoS

perceived by customers for the period under review.

The drive test was designed to be representative of the

population relative to the traffic of the network.

Measurements were scheduled to reflect accurately the

traffic variations over the hours of a day, and user's

behavior.6

Call Type and

Window Call

Long calls and Short calls were used.

Voice Telephony was tested in Mobile Originating Call

(MOC) direction. The following call durations were used:

CD1: 10 seconds for call setup testing;

CD2: 120 seconds for typical tests, default call duration;

Call Window: Call Duration + 30 seconds, (for the setup

and release phases) + 30 seconds (for the minimum

pause. Interval), for the default call duration this results

in 180 seconds.7

Equipment

Test equipment used : TEMS Symphony 7.7 using

Samsung Galaxy Note S4 (SM-N910F), Dell Latitude with

Probe Controller v1.9.2

KPI8

The measurements focused on the following network

parameters:

(i) Dropped Call Ratio (DCR)

(ii) Call Set-up Success Ratio (CSSR)

Log files

The log files for each test case were stored in a different

folder with unique names. The log files were recorded per

network operator.

5 Section 6.3 of ETSI TS 102 250-4 V.1.1.2 (2003-10) 6 Section 4.8 of ETSI EG 202 057-3 V1.1.1 (2005-04) 7 Section 4.2.1 of ETSI TS 102 250-5 V1.2.1 (2005-05) 8 End-User and Subscriber Service Charter Regulations of 2016

8 | P a g e

Mobile terminal

used

The test calls were terminated on each operator’s test

platform or IVR system.

Network tested Cell-C, MTN, Vodacom and Telkom.

Sim card Test sim cards provided by each operator were used during

the drive test.

Vehicle used

Toyota Hilux Double Cab equipped with drive test equipment

was used. All the road traffic rules were observed during the

drive test. The speed was maintained at an average of 60km

in town, built-up areas and 100km/h on highways.

2.4 Statistical Significance

Statistical analysis is used to ensure that collected samples is representative in the

reported KPIs results. This means that one can be “relatively sure” that the results

represent the reality (true population mean) and that they did not occur by chance.

Statistical theory provides tools to assess the statistical significance of measurement

observations with a given sample count and standard deviation.

2.5 Measurement Parameters and Targets

2.5.1 Parameter Targets

The Average Drop Call Ratio should be less than 3% over 6 months period and

Average Call Setup Success Ratio must be greater than 98% over six months period

as per the “End-User and Subscriber Service Charter Regulations of 2016”.

2.5.2 Drop-Call Ratio

Dropped Call Ratio (DCR) is the proportion of incoming and outgoing calls, which,

once correctly established and therefore having been assigned a traffic channel, is

dropped or interrupted prior to the deliberate completion by the user.9

The formula to calculate DCR is shown below:

DCR = D/S*100,

Where D represents the number of dropped calls and S is the number of successful

call attempts.

9 End-User and Subscriber Service Charter Regulations of 2016

9 | P a g e

2.5.3 The Call Setup Success Ratio

The Call setup Success Ratio (CSSR) is the percentage of calls that are successfully

setup as a percentage of the total call attempts.10

The formula to calculate CSSR is shown below:

CSSR = Y/X *100,

Where Y represents the calls that are successfully setup and X is the total call

attempts.

10 End-User and Subscriber Service Charter Regulations of 2016

10 | P a g e

3 Results and Analysis

This section provides a summary of the mobile operator’s performance results based

on the drive test route in the following test area: Ga-Riba/Ga-Nkoana, Tzaneen,

Hoedspruit, Modjadjiskloof and Bela-Bela.

3.1 Accessibility and Retainability Measurements

Table 3: Summary of Results

Route Name Operator Call Setup

Success Ratio (%)

Drop Call Ratio (%)

Ga-Nkoana/Ga-Riba

Cell-C 96.96% 4.61%

Vodacom 93.53% 1.36%

MTN 93.74% 0.67%

Telkom 97.42% 6.92%

Tzaneen

Cell-C 95.46% 1.94%

Vodacom 98.31% 2.99%

MTN 96.89% 1.78%

Telkom 98.21% 10.76%

Hoedspruit

Cell-C 81.74% 3.48%

Vodacom 92.41% 6.30%

MTN 98.46% 0.69%

Telkom 92.89% 10.33%

Modjadji

Cell-C 96.60% 1.10%

Vodacom 95.65% 1.97%

MTN 96.68% 1.84%

Telkom 95.53% 12.13%

Bela-Bela

Cell-C 97.26% 2.24%

Vodacom 98.80% 1.21%

MTN 96.15% 2.95%

Telkom 93.07% 5.15%

Provincial

Cell-C 93.72% 2.65%

Vodacom 95.83% 2.60%

MTN 96.40% 1.56%

Telkom 95.38% 8.98%

Table 3 above shows Voice Call measurement results in each route at provincial level

for the all the operators. Additional KPIs are presented in the Appendix A.

11 | P a g e

3.1.1 Call Setup Success Ratio (CSSR)

Figure 2: Call Setup Success Ratio (CSSR) per Route

Figure 2 result shows that Cell C did not comply with the 98% CSSR target in all the

target routes. Vodacom complied with the target in Tzaneen and Bela-Bela route but

failed to comply with the target on the all other three routes. MTN passed the target

in Hoespruit but failed the target in the other four areas.

Figure 3: Total Call Setup Success Ratio with Statistical Significance

MTN Provincial Call Setup Success Rate (CSSR) was the highest followed by Vodacom,

Telkom and Cell C in the respective descending order. None of the operators complied

with the ICASA’s 98% CSSR target as per “End-User and Subscribers Service charter

regulations of 2016”. Vodacom, MTN and Telkom Provincial CSSR differences are not

80.00%

85.00%

90.00%

95.00%

100.00%

Ga-Riba/Ga-Nkoana Tzaneen Hoedspruit Modjadjiskloof Bela-Bela

Call Setup Success Ratio per Route

CellC Vodacom MTN Telkom Target

95.8 % 96.4 %

93.7 %95.4 %

90%

92%

94%

96%

98%

100%

Vodacom MTN Cell C Telkom

Call Setup Success Rate

12 | P a g e

statistically significant. CellC CSSR differences are not statistically significant from

Vodacom, MTN and Telkom.

3.1.2 Drop Call Ratio (DCR)

Figure 4: Drop Call Ratio (DCR) per Route

Figure 4 result shows that MTN complied with the 3% DCR target in all the target

routes. Vodacom complied with the target on four target routes (Ga-Riba/Ga-Nkoana,

Tzaneen, Modjadjiskloof and Bela-Bela) but failed at Hoedspruit route. CellC complied

with the target in three areas (Tzaneen, Modjadjiskloof and Bela-Bela) but failed the

target in Ga-Riba/Ga-Nkoana and Hoedspruit.

Figure 5: Total Call Drop Ratio with Statistical Significance

-1.00%

4.00%

9.00%

14.00%

Ga-Riba/Ga-Nkoana Tzaneen Hoedspruit Modjadjiskloof Bela-Bela

Limpopo Province Call Drop Ratio

CellC Vodacom MTN Telkom Target

2.60% 1.56% 2.65% 8.98%0%

2%

4%

6%

8%

10%

12%

Vodacom MTN Cell C Telkom

Call Drop Rate

13 | P a g e

MTN provincial Call Drop Rate (CDR) is the lowest followed by Vodacom, Cell C and

Telkom respectively. Vodacom, MTN and CellC complied with ICASA’s 3% CDR target.

Telkom did not comply with the CDR target. CDR differences are statistically

significant between Telkom and other operators (Vodacom, Cell C and MTN). The

difference between Vodacom, MTN and Cell C is not statistically significant.

3.2 Technology Distribution

Figure 6: Technology Distribution

Cell C calls were on WCDMA most of the time, while the other operator calls were on

GSM for more than 50% of the time.

4 Conclusion

This section provides the summary of the key findings of all measurements. The

results illustrate a snapshot of the mobile network performance and customer

experience within the measured time and location context.

The results indicate that the End-Users Quality of Service and operators’ network

performance varies significantly on per location basis. In most cases, all operators

performed better, and delivered an acceptable Quality of Service to End-Users in

densely populate areas than in sparsely populated rural areas and major roads.

As we benchmark the Operators, the results indicate that in terms of Call Setup

Success Rate Vodacom, MTN and Telkom performance is very close to each other,

while in terms of Drop Call Rate performance Vodacom, MTN and Cell C were very

close to each other. Telkom needs to improve its Drop Call Rate as shown by the

results.

Cell-C Vodacom MTN Telkom

27.77%

85.39%

58.00% 56.41%

70.80%

12.37%

41.58% 41.16%

1.43% 2.14% 0.42% 2.43%

TECHNOLOGY DISTRIBUTION

GSM WCDMA NoService

14 | P a g e

15 | P a g e

5 Appendixes

5.1 Appendix A: Drive Test Results KPI’s

Table 4: Call Setup Success Ratio (CSSR) per Route

Phase 1 Phase 2 Total

Route Name

Op

era

tor

Ca

ll

Att

em

pt

Ca

ll S

etu

p

CS

SR

(%

)

Ca

ll

Att

em

pt

Ca

ll S

etu

p

CS

SR

(%

)

Ca

ll

Att

em

pt

Ca

ll S

etu

p

CS

SR

(%

)

Ga-Nkoana

Cell-C 276 269 97.46% 546 528 96.70% 822 797 96.96%

Vodacom 438 410 93.61% 551 515 93.47% 989 925 93.53%

MTN 351 346 98.58% 591 537 90.86% 942 883 93.74%

Telkom 494 480 97.17% 437 427 97.71% 931 907 97.42%

Tzaneen

Cell-C 444 429 96.62% 460 434 94.35% 904 863 95.46%

Vodacom 467 460 98.50% 477 468 98.11% 944 928 98.31%

MTN 477 459 96.23% 487 475 97.54% 964 934 96.89%

Telkom 424 416 98.11% 412 405 98.30% 836 821 98.21%

Hoedspruit

Cell-C 426 357 83.80% 412 328 79.61% 838 685 81.74%

Vodacom 454 433 95.37% 468 419 89.53% 922 852 92.41%

MTN 486 481 98.97% 489 479 97.96% 975 960 98.46%

Telkom 430 396 92.09% 428 401 93.69% 858 797 92.89%

Modjadji

Cell-C 444 426 95.95% 439 427 97.27% 883 853 96.60%

Vodacom 453 439 96.91% 305 286 93.77% 758 725 95.65%

MTN 456 440 96.49% 448 434 96.88% 904 874 96.68%

Telkom 427 407 95.32% 400 383 95.75% 827 790 95.53%

Bela-Bela

Cell-C 473 463 97.89% 441 426 96.60% 914 889 97.26%

Vodacom 493 487 98.78% 593 586 98.82% 1086 1073 98.80%

MTN 457 438 95.84% 504 486 96.43% 961 924 96.15%

Telkom 457 426 93.22% 510 474 92.94% 967 900 93.07%

16 | P a g e

Table 5: Drop Call Ratio (DCR) per Route

Phase 1 Phase 2 Total

Route Name

Op

era

tor

Ca

ll A

ttem

pt

Ca

ll

Es

tab

lis

he

d

Ca

ll D

rop

pe

d

DC

R (

%)

Ca

ll A

ttem

pt

Ca

ll

Es

tab

lis

he

d

Ca

ll D

rop

pe

d

DC

R (

%)

Ca

ll A

ttem

pt

Ca

ll

Es

tab

lis

he

d

Ca

ll D

rop

pe

d

DC

R (

%)

Ga-Riba/Ga-Nkoana

Cell-C 127 121 6 4.96% 167 161 7 4.35% 294 282 13 4.61%

Vodacom 131 130 2 1.54% 166 164 2 1.22% 297 294 4 1.36%

MTN 139 137 1 0.73% 176 161 1 0.62% 315 298 2 0.67%

Telkom 133 126 13 10.32% 169 163 7 4.29% 302 289 20 6.92%

Tzaneen

Cell-C 135 127 2 1.57% 142 131 3 2.29% 277 258 5 1.94%

Vodacom 166 164 2 1.22% 135 131 5 3.82% 301 295 7 2.37%

MTN 142 140 3 2.14% 145 141 2 1.42% 287 281 5 1.78%

Telkom 141 121 19 15.70% 143 130 15 11.54% 284 251 34 13.55%

Hoedspruit

Cell-C 134 98 4 4.08% 135 103 3 2.91% 269 201 7 3.48%

Vodacom 130 119 8 6.72% 126 119 7 5.88% 256 238 15 6.30%

MTN 145 143 1 0.70% 147 146 1 0.68% 292 289 2 0.69%

Telkom 121 108 9 8.33% 124 105 13 12.38% 245 213 22 10.33%

Modjadjiskloof

Cell-C 136 135 3 2.22% 141 138 0 0.00% 277 273 3 1.10%

Vodacom 129 123 1 0.81% 137 131 4 3.05% 266 254 5 1.97%

MTN 140 136 3 2.21% 141 136 2 1.47% 281 272 5 1.84%

Telkom 131 117 12 10.26% 137 122 17 13.93% 268 239 29 12.13%

Bela-Bela

Cell-C 163 151 0 0.00% 307 248 5 2.02% 470 399 5 1.25%

Vodacom 161 159 3 1.89% 191 187 1 0.53% 352 346 4 1.16%

MTN 161 152 6 3.95% 141 127 2 1.57% 302 279 8 2.87%

Telkom 122 109 5 4.59% 161 140 7 5.00% 283 249 12 4.82%

17 | P a g e

5.2 Appendix B: Coverage Maps

5.2.1 Accessibility: Signal Levels for Serving Cells 5.2.1.1 Vodacom

GSM WCDMA

Figure 7: Limpopo Vodacom GSM and WCDMA Signal Levels

5.2.1.2 MTN

GSM WCDMA

Figure 8: Limpopo MTN GSM and WCDMA Signal Levels

5.2.1.3 Cell-C

GSM WCDMA

Figure 9: Limpopo Cell C GSM and WCDMA Signal Levels

18 | P a g e

5.2.1.4 Telkom

GSM WCDMA

Figure 10: Limpopo Telkom GSM and WCDMA Signal Levels

5.2.2 Retainability: Signal Levels for Serving Cells 5.2.2.1 Vodacom

GSM WCDMA

Figure 11: Limpopo Vodacom GSM and WCDMA Signal Levels

5.2.2.2 MTN

GSM WCDMA

Figure 12: Limpopo MTN GSM and WCDMA Signal Levels

19 | P a g e

5.2.2.3 Cell-C

GSM WCDMA

Figure 13: Limpopo Cell C GSM and WCDMA Signal Levels

5.2.2.4 Telkom

GSM WCDMA

Figure 14: Limpopo Telkom GSM and WCDMA Signal Levels

5.2.3 Serving Technology Details

Vodacom MTN

Cell-C Telkom

Figure 15: Limpopo Serving Technology Plot

20 | P a g e