1x evdo optimization_guide_v3.0_chr

Nortel Networks Confidential and Proprietary

. . . . . . . . . .

. . . . . .. . . . Core RF Engineering Dept. 5395

1xEV-DO RF Optimization Guide

1xEV – DO Release 3.x

Product Line: 1xEV-DO Release 3.0 ChR Version: Initial Release Issue Date: March 3, 2006 Author: Mike Woodley Department: Core RF Authorizing Manager: Frank Jager / Farhad Bassirat 2003 Nortel Networks All rights reserved. Information subject to change without notice. The information disclosed herein is proprietary to Nortel Networks or others and is not to be used by or disclosed to unauthorized persons without the written consent of Nortel Networks. The recipient of this document shall respect the security status of the information.

Nortel Networks Confidential and Proprietary 2

Table of Contents

TABLE OF CONTENTS............................................................................................................................. 2

1 INTRODUCTION ................................................................................................................................ 8 1.1 RELATED DOCUMENTS.................................................................................................................... 8 1.2 SCOPE.............................................................................................................................................. 9 1.3 REVISION HISTORY ......................................................................................................................... 9 1.4 CONTRIBUTORS ............................................................................................................................... 9 1.5 AUDIENCE ....................................................................................................................................... 9

2 1XEVDO OPTIMIZATION EVENTS FLOW CHART................................................................. 10

3 1XEV-DO PRE OPTIMIZATION OBJECTIVES.......................................................................... 11 3.1 ENTRANCE AND EXIT CRITERIA.................................................................................................... 11

3.1.1 RF Design Review ................................................................................................................ 11 3.1.2 Coverage Control ................................................................................................................. 11 3.1.3 RF Coverage Availability ..................................................................................................... 11 3.1.4 Inter-RNC Border Design..................................................................................................... 13 3.1.5 CIQ Review and Provisioning Verification .......................................................................... 15

4 1XEV-DO OPTIMIZATION EVENTS............................................................................................ 17 4.1 INITIAL RF PARAMETERS VERIFICATION...................................................................................... 17 4.2 PRE OPTIMIZATION SYSTEM HEALTH CHECK................................................................................ 17 4.3 FTAP AND RTAP TESTING ........................................................................................................... 19 4.4 TCP / IP PARAMETERS .................................................................................................................. 20 4.5 SHAKEDOWNS ............................................................................................................................... 24 4.6 GOLDEN VALUE (STATIONARY) TESTING...................................................................................... 24 4.7 CLUSTER DRIVE TESTING.............................................................................................................. 25 4.8 TROUBLESHOOTING....................................................................................................................... 25

5 DATA COLLECTION / POST PROCESSING TOOL SETUP .................................................... 26 5.1 LAPTOP SETUP............................................................................................................................... 26 5.2 AT SETUP...................................................................................................................................... 27 5.3 DATA COLLECTION TOOL SETUP................................................................................................... 27

5.3.1 RNC Logging Setup .............................................................................................................. 27 5.3.2 Optis-M Setup ....................................................................................................................... 32 5.3.3 Optis-M Log Masks .............................................................................................................. 35 5.3.4 Invex3G Chassis Setup ......................................................................................................... 38 5.3.5 InVex3G Log Masks.............................................................................................................. 40

5.4 AT LOG DATA POST-PROCESSING TOOL SETUP............................................................................ 41 5.4.1 RF Optimizer EV-DO ........................................................................................................... 41 5.4.2 XCAP-DO............................................................................................................................. 43

5.5 DOM - RNC LOG POST PROCESSING TOOLS................................................................................. 45 5.5.1 Converting RNC Logs from Binary to Text........................................................................... 45 5.5.2 NEWTUN (Post Processing and Analysis of RNC Logs)...................................................... 45

5.6 OM DATA COLLECTION TEMPLATE CONFIGURATION SETUP........................................................ 54 5.7 CELLDM DATA COLLECTION ...................................................................................................... 56 5.8 OM ANALYSIS TOOLS ................................................................................................................... 61 5.9 TCP PERFORMANCE LOGGING TOOLS........................................................................................... 63

6 SHAKEDOWNS................................................................................................................................. 65 6.1 OBJECTIVES................................................................................................................................... 65 6.2 EVDO SITE SHAKEDOWN PROCESS .............................................................................................. 65


6.3 EVDO INTER SUBNET ACTIVE HANDOFF (ISSHO) SHAKEDOWN PROCESS .................................. 69 6.4 (A13) DORMANT SESSION HANDOFF SHAKEDOWN PROCESS ........................................................ 71 6.5 DATA PROCESSING AND ANALYSIS ............................................................................................... 73

6.5.1 EVDO Site Shakedown Analysis ........................................................................................... 73 6.5.2 Inter RNC Active Handoff Shakedown Analysis ................................................................... 77 6.5.3 A13 Dormant Session Handoff Analysis ............................................................................... 78

7 GOLDEN VALUE (STATIONARY) TESTING ............................................................................. 79 7.1 INTRODUCTION.............................................................................................................................. 79 7.2 FTP SERVER CONFIGURATION ...................................................................................................... 79 7.3 GOLDEN VALUE TESTING PROCESS............................................................................................... 79 7.4 DATA PROCESSING AND ANALYSIS ............................................................................................... 82

8 OM COLLECTION AND ANALYSIS ............................................................................................ 84 8.1 CONNECTION ATTEMPTS ............................................................................................................... 84 8.2 FAILED CALL ATTEMPTS (FCA / ACCESS FAILURE)...................................................................... 84 8.3 DROPPED CONNECTIONS (ABNORMAL CONNECTION CLOSES)...................................................... 84 8.4 SESSION SETUP PERFORMANCE (INCLUDES A13 SESSION HANDOFFS) ........................................... 85

8.4.1 UATI Request Initiated A13 Dormant Handoff Performance............................................... 85 8.4.2 Prior Session Initiated Dormant Handoff Performance ....................................................... 85 8.4.3 A10 Connection Setup Performance..................................................................................... 85 8.4.4 Session Configuration Failures ............................................................................................ 86

8.5 SERVING SECTOR SWITCHING AND SOFT/SOFTER HANDOFF PERFORMANCE ................................ 86 8.6 INTER RNC ACTIVE HANDOFF TRAFFIC STATISTICS AND SECONDARY BORDER MEASUREMENT. 86 8.7 1XEVDO PAGING PERFORMANCE MEASUREMENTS ..................................................................... 86 8.8 PER SECTOR AIRLINK RESOURCES ALLOCATION (BLOCKING) ...................................................... 86

9 CLUSTER DRIVE TESTING........................................................................................................... 87 9.1 DATA COLLECTION PROCESS ........................................................................................................ 87 9.2 DATA PROCESSING AND ANALYSIS ............................................................................................... 92

9.2.1 Access Failure Rate .............................................................................................................. 92 9.2.2 Connection Drop Rate .......................................................................................................... 95 9.2.3 Throughput in Mobility Condition........................................................................................ 99 9.2.4 Connection Setup Time....................................................................................................... 102

10 TROUBLESHOOTING............................................................................................................... 103 10.1 SESSION SETUP AND A13 HANDOFF SUCCESS RATE ................................................................... 103

10.1.1 Data Collection Methods.................................................................................................... 103 10.1.1.1 AT Logs..................................................................................................................................... 103 10.1.1.2 RNC Logs .................................................................................................................................. 103 10.1.1.3 OMs ........................................................................................................................................... 109

10.1.2 KPI Calculation.................................................................................................................. 113 10.1.2.1 KPI Value and Tolerance Factor................................................................................................ 113 10.1.2.2 Calculating KPI from AT Logs.................................................................................................. 114 10.1.2.3 Calculating KPI from RNC Logs............................................................................................... 114 10.1.2.4 Calculating KPI from OMs........................................................................................................ 114

10.1.3 Troubleshooting Guide ....................................................................................................... 115 10.2 SESSION SETUP TIME................................................................................................................... 118




10.2.3 Troubleshooting Guide ....................................................................................................... 120 10.3 CONNECTION SETUP TIME........................................................................................................... 120



10.3.3 Troubleshooting Guide ....................................................................................................... 122 10.4 ACCESS FAILURES ....................................................................................................................... 122



10.4.3 Troubleshooting Guide ....................................................................................................... 128 10.5 CALL BLOCKS ............................................................................................................................. 133

10.5.1 Data Collection Methods.................................................................................................... 133 10.5.1.1 AT Logs..................................................................................................................................... 133 10.5.1.2 RNC Logs .................................................................................................................................. 133 10.5.1.3 Blocking OMs............................................................................................................................ 134

10.5.2 Blocking KPI Calculation................................................................................................... 134 10.5.2.1 KPI Value and Tolerance Factor................................................................................................ 134 10.5.2.2 Calculating KPI from AT Logs.................................................................................................. 134 10.5.2.3 Calculating KPI from RNC Logs............................................................................................... 134 10.5.2.4 Calculating KPI from OMs........................................................................................................ 134

10.5.3 Troubleshooting Guide ....................................................................................................... 135 10.6 CONNECTION DROPS ................................................................................................................... 138

10.6.1 Data Collection Methods.................................................................................................... 138 10.6.1.1 AT Logs..................................................................................................................................... 138 10.6.1.2 RNC Logs .................................................................................................................................. 139 10.6.1.3 Dropped Connection OMs ......................................................................................................... 140

10.6.2 Dropped Connection KPI Calculation ............................................................................... 141 10.6.2.1 KPI Value and Tolerance Factor................................................................................................ 141 10.6.2.2 Calculating KPI from AT Logs.................................................................................................. 141 10.6.2.3 Calculating Dropped Connections KPI from RNC Logs ........................................................... 142 10.6.2.4 Calculating Dropped Connection KPI from OMs...................................................................... 142

10.6.3 Dropped Connection Troubleshooting Guide..................................................................... 143 10.7 FL SECTOR SWITCHING SUCCESS RATE ...................................................................................... 154


10.7.2 Forward Sector Switching Success Rate KPI Calculation ................................................. 154 10.7.2.1 KPI Value and Tolerance Factor................................................................................................ 154 10.7.2.2 Calculating KPI from AT Logs.................................................................................................. 154 10.7.2.3 Calculating KPI from RNC Logs............................................................................................... 154 10.7.2.4 Calculating KPI from OMs........................................................................................................ 154

10.7.3 Troubleshooting Guide ....................................................................................................... 155 10.8 FL SECTOR SWITCHING TIME...................................................................................................... 155

10.8.1 Data Collection Methods.................................................................................................... 155 10.8.1.1 AT Logs..................................................................................................................................... 155 10.8.1.2 RNC Logs .................................................................................................................................. 155


10.8.1.3 OMs ........................................................................................................................................... 156 10.8.2 KPI Calculation.................................................................................................................. 156

10.8.2.1 KPI Value and Tolerance Factor................................................................................................ 156 10.8.2.2 Calculating KPI from AT Logs.................................................................................................. 156 10.8.2.3 Calculating KPI from RNC Logs............................................................................................... 156 10.8.2.4 Calculating KPI from OMs........................................................................................................ 157

10.8.3 Troubleshooting Guide ....................................................................................................... 157 10.9 RL SHO SUCCESS RATE.............................................................................................................. 157



10.9.3 Troubleshooting Guide ....................................................................................................... 159 10.10 FL PER-USER THROUGHPUT ................................................................................................... 160

10.10.1 Data Collection Methods................................................................................................ 160 10.10.1.1 AT Logs..................................................................................................................................... 160 10.10.1.2 TCP Trace Logs......................................................................................................................... 160 10.10.1.3 RNC Logs .................................................................................................................................. 162 10.10.1.4 OMs ........................................................................................................................................... 162

10.10.2 KPI Calculation.............................................................................................................. 162 10.10.2.1 KPI Value and Tolerance Factor................................................................................................ 162 10.10.2.2 Calculating KPI from AT Logs.................................................................................................. 162 10.10.2.3 Calculating KPI from RNC Logs............................................................................................... 163 10.10.2.4 Calculating KPI from OMs........................................................................................................ 163

10.10.3 Troubleshooting Guide ................................................................................................... 163 10.11 FL PER-SECTOR THROUGHPUT................................................................................................ 180

10.11.1 Data Collection Methods................................................................................................ 180 10.11.1.1 AT Logs..................................................................................................................................... 180 10.11.1.2 RNC Logs .................................................................................................................................. 180 10.11.1.3 OMs ........................................................................................................................................... 180


10.11.3 Troubleshooting Guide ................................................................................................... 181 10.12 RL PER-USER THROUGHPUT ................................................................................................... 181



10.12.3 Troubleshooting Guide ................................................................................................... 183 10.13 RL PER-SECTOR THROUGHPUT ............................................................................................... 185


10.13.2 KPI Calculation.............................................................................................................. 185 10.13.2.1 KPI Value and Tolerance Factor................................................................................................ 185 10.13.2.2 Calculating KPI from AT Logs.................................................................................................. 185


10.13.2.3 Calculating KPI from RNC Logs............................................................................................... 185 10.13.2.4 Calculating KPI from OMs........................................................................................................ 185

10.13.3 Troubleshooting Guide ................................................................................................... 186 10.14 FL ACHIEVABLE DATA RATE.................................................................................................. 186



10.14.3 Troubleshooting Guide ................................................................................................... 187 10.15 RL ACHIEVABLE DATA RATE ................................................................................................. 191



10.15.3 Troubleshooting Guide ................................................................................................... 192 10.16 PACKET LATENCY ................................................................................................................... 193



10.16.3 Troubleshooting Guide ................................................................................................... 194 10.17 DO-TO-1X HANDOFF DELAY .................................................................................................. 194



10.17.3 Troubleshooting Guide ................................................................................................... 195 10.18 DO-TO-1X SUCCESS RATE ...................................................................................................... 195



10.18.3 Troubleshooting Guide ................................................................................................... 196 APPENDIX A: EVDO OPTIMIZATION AND TROUBLESHOOTING FLOW CHARTS............ 197


APPENDIX B: CLI SYSTEM READINESS CHECK.......................................................................... 202

APPENDIX C: SHAKEDOWN FORM ................................................................................................. 205

APPENDIX D: EXAMPLES OF PAST ISSUES................................................................................... 206


1 Introduction

This document provides a guideline for RF engineers to do the RF optimization of a 1xEV-DO system. 1xEV-DO system is optimized to carry non real-time packet data services and hence does not support voice services. In the forward link, 1xEV-DO air interface is designed to provide up to 2.4 Mbps data rate with only one 1.25 MHz carrier using TDM (Time Division Multiplexing) system. In the reverse link, 1xEV-DO provides up to 153.6 kbps data rate similar to 1xRTT. Since the forward link physical layer implementation of 1xEV-DO and CDMA/1xRTT are different, they cannot exist in the same frequency. 1xEV-DO has to be deployed in a separate carrier.

1.1 Related Documents

• [1] “1xEV-DO Coverage and Capacity Performance,” v2.0, Muhieddin Najib, Core RF Engineering, February 2004 (http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/rf/lb)

• [2] “1xEV-DO Provisioning Guideline,” v2.1.03, Kenneth Ho, Core Network Engineering, August 2004 (http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/dim/gl)

• [3] “1xEV-DO RF Parameters and Datafill Guide,” v3.01, Mike Garramone, Core RF Engineering, October 2005. (http://livelink.us.nortel.com/livelink/livelink.exe?func=ll&objId=9655009&objAction=browse&sort=name)

• [4] “DOM-RNC Logging,” Wei Lou, Core RF Engineering, draft to be released. • [5] “QTP-5500 Access Terminal Users Guide,” 80-B1311-2, Rev A, Qualcomm, March 2004. • [6] “1xRTT and 1xEV-DO RF Link Budgets,” Muhieddin Najib, Core RF Engineering, October

2004 (presentation to Orange). • [7] “1xEV-DO RF System Performance,” Miro Budic, Core RF Engineering, October 2004

(presentation to Orange). • [8] “1xEV-DO RF Parameters,” Brian Troup, Core RF Engineering, September 2004

(presentation in Verizon Users Group). • [9] “Inter-RNC Active Handoff RF Engineering Guideline,” Version 0.02, Brian Troup, October

2005. (http://livelink.us.nortel.com/livelink/livelink.exe?func=ll&objId=9655009&objAction=browse&sort=name)

• [10] “Optis-M Quick Start Guide,” v1.05, WirelessLogix, 2004 [11] “OPTis-M User Guide,” v1.05, WirelessLogix, 2004

• [11] “Maximum 1xEV-DO Application Layer Throughput,” Dept. 2N51, October 2002. • [12] “Generic Network Acceptance Process – 1xEVDO Network,” v4.1, Mike Woodley, January

2005. • [13] “Microsoft Windows 2000 TCP/IP Implementations Details,” Dave MacDonald and Warren

Barkley, 2005 Microsoft Corp. • [14] “RFC 3481 – TCP over Second (2.5G) and Third (3G) Generation Wireless Networks,”

www.faqs.org/rfcs/rfc3481.html • [15] “TAP Users Guide”, Rev. A, Airvana, April 22, 2001 • [16] “1xEVDO Troubleshooting Guide”, Std 2.05, NTP 411-2133-949, April 2005 • [17] “1xEVDO Performance Measurements”, Draft 4.07, NTP 411-2133-924, September 2005. • [18] “Invex3G Quick Start Guide”, version 4.3, Andrew Corporation, 2005. • [19] “1xEV-DO CELLDM Logging Feature Application Guide”, Wei Lou, Version 0.1, October

20, 2005. • [20] “A13 Dormant Handoff and UA10 Minimization Application Guide”, Miroslav Budic, Draft,

September 23, 2005.


1.2 Scope This document only covers the RF optimization aspect of 1xEV-DO Release 0. RF optimization guideline document for IS-95 CDMA is available as NTP 411-2133-004. RF optimization guideline document for IS-2000 (1xRTT) is available at http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/po/opt.

1.3 Revision History

Issue Date Reason 1.0 November 4, 2005 Initial release for EVDO rel. 3.0. 1.1 March 3, 2006 Updated to reflect EVDO 3.0 ChR.

The latest version of this document is available at http://navigate.us.nortel.com/imds?pg=/eng/wne/cdma/sys/po/opt.

1.4 Contributors The following are the key contributors to the contents of the document:

• Rubianto Satrio • Wei Lou • Brian Troup • Miro Budic • Martin Kendall • Mike Woodley • Alexander Contreras • Saied Kazeminejad • Gordon Edwards • Remo Agostino

1.5 Audience This document is intended for Nortel Networks and customer RF engineers that are involved in the 1xEV-DO deployment and want to understand how to optimize a 1xEV-DO network. A basic understanding of IS95 and IS2000, as well as general RF principles is expected of the reader. THIS VERSION IS CURRENTLY FOR INTERNAL AUDIENCES ONLY


2 1xEVDO Optimization Events Flow Chart

N

Y

Y

N

Establish Network Optimization Entrance and Exit Criteria (Section 3.1)

1xEVDO / CDMA 1xRTT network RF Coverage Design Review

(Section 3.1.1 – 3.1.4)

Coverage design is sufficient to meet optimization exit criteria?

Review Customer CIQ responses and current network provisioning

(Section 3.1.5)

Is network provisioning and call models sufficient to support optimization exit

criteria?

RF Datafill Parameters Review and Verification (Section 4.1)

Pre Optimization System Health Check (Section 4.2 and Appendix B)

Pre O

ptimization O

bjectivesFTAP and RTAP Airlink Bandwidth

Verification (Section 4.3)

TCP / IP Parameter Review and Verification

(Section 4.4)

Site and Border Shakedowns (Section 6)

Golden Value (Stationary Testing) (Section 7)

Cluster Drive Testing / Optimization (Section 8 - 9)

Exit Criteria Satisfied?

Perform Troubleshooting (Section 10)

Exit Market Y

In Market O

ptimization Activities


3 1xEV-DO Pre Optimization Objectives

This chapter gives an overview of the events and processes in the 1xEV-DO RF optimization work. The subsequent chapters (Chapter 3 onward) will then discuss them in more detail.

3.1 Entrance and Exit Criteria Entrance Criteria

In general there are four important steps that need to be done prior to the 1xEV-DO RF optimization: • Review the 1xEV-DO RF design (in iPlanner) to make sure that it meets the coverage and

capacity/throughput objectives. • Check that the 1xEV-DO network design matches the requirements listed in the CIQ (Customer Input

Questionnaire) response. • Ensure that the recommended datafill is used for the initial RF parameters. • 1xEV-DO basic system readiness checkup is conducted. • The 1xEV-DO drive test routes are determined and agreed upon. • Exclusion criteria and warranty boundaries agreed upon by Nortel and the customer

Exit Criteria

The detailed exit criteria need to be negotiated between Nortel Networks and the customer. They have to be agreed upon prior to the 1xEV-DO RF optimization effort. Metrics that can be used for this purpose include: • Data-rate-to-DRC matching (i.e., showing that the decoded data rate received matches the DRC

requested). • Single-user application throughput under a certain RF and mobility condition. • Access failure rate. • Connection drop rate. • Connection setup time.

3.1.1 RF Design Review Just like in CDMA, a good 1xEV-DO network starts with a good 1xEV-DO RF design. Therefore, the first step in 1xEV-DO RF optimization should be to review the final RF design. Some important items to check are discussed below.

3.1.2 Coverage Control One of the most important principles of CDMA RF design, i.e. coverage control, still applies to 1xEV-DO. In 1xEV-DO, the AT (Access Terminal) continuously estimates the SINR (signal-to-interference-plus-noise ratio) in the forward link and requests a data rate based on the SINR. In general, the better the SINR, the higher the data rate requested (the exact algorithm depends on the AT vendor implementation). Hence we should check the RF design to ensure that each sector covers only the area it is intended to cover. Any potential pilot pollution area should be cleaned up as much as possible, e.g., by using different antenna configuration. The final RF design result should show that: • The forward link SINR or Ec/Nt (C/I) is ≥ -2 dB throughout the network.

3.1.3 RF Coverage Availability As in CDMA, the 1xEV-DO RF design must satisfy the path loss requirement for the designed data rate from the link budget. This can be shown using the pilot composite coverage plot or the required AT Transmit power plot.


The 1xEV-DO cell radius depends, among other things, on the designed data rate. If the 1xEV-DO network is overlaid 1-to-1 on an IS-95 network, then it is good for the forward link data rate of 307 kbps (at 1900 MHz and with AT receive diversity) and reverse link data rate of 19.2 kbps as the chart below shows [1]. If higher data rate is needed, then additional cell sites are needed.

38.4 kbps

76.8 kbps

153.6 kbps

307.2 kbps

614.4 kbps

921.6 kbps

1228.8 kbps

1843.2 kbps

2457.6 kbps

w/o Diversity 136% 111% 99% 88% 73% 61% 55% 45% 40% 1900

MHz w/ Diversity 161% 131% 116% 104% 87% 72% 65% 53% 47%

w/o Diversity 158% 129% 114% 102% 85% 71% 64% 52% 46% 850

MHz w/ Diversity 185% 151% 134% 120% 100% 83% 75% 62% 54%

w/o Diversity 145% 118% 105% 93% 78% 65% 59% 48% 42% 450

MHz w/ Diversity 169% 137% 122% 109% 91% 76% 68% 56% 49%

1xEV-DO forward link cell radius relative to IS-95 EVRC.

9.6 kbps 19.2 kbps 38.4 kbps 76.8 kbps 153.6 kbps

All Frequencies

111% 101% 89% 75% 57%

1xEV-DO reverse link cell radius relative to IS-95 EVRC.

Another important thing to consider is that a certain data rate in the reverse link is required to support the high data rate in the forward link. Therefore the RF design has to ensure the reverse link data rate is enough to support the forward link data rate. The chart below shows the measurement results from LAV network in Ottawa [6], [7]. To support the TCP ACKs, the reverse link data rate has to be at least 1/40 of the forward link data rate. So, for example, a forward link data rate of 2 Mbps requires a reverse link data rate of 50 kbps or higher.

Max Fwd Rate vs Rev Rate

80828486889092949698

100

19.2 38.4 76.8

Reverse Rate [kbps]

Nor

mal

ized

Fw

d R

ate

[%]

Reverse link data rate impact on forward link throughput.


3.1.4 Inter-RNC Border Design In Nortel’s EVDO release 3.0 there is a new feature called “Inter RNC Active Handoff” (IRAHO) that will allow ATs to add pilots to its Active set from DOMs primarily homed on adjacent RNCs. The intent of the feature is to:

• Prevent dropped connections and access failures in border areas • Enhance sector and AT throughput in border areas

This feature allows the AT to add pilots from a DOM in another subnet through secondary homing. Secondary homing allows an operator to home DOMs to up to 8 secondary RNCs in addition to its Primary RNC (i.e. data fill the routing table in the DOM with the primary RNC and up to 8 secondary RNC’s). Through secondary homing an AT can add pilots from DOMs in an adjacent subnet without having any knowledge of that subnet’s Color Code, or Subnet ID. The figure below shows an example of the physical connectivity of DOMs and RNCs in a multi homed scenario.

Figure – EVDO Multi Homing Configuration example The limitations of this feature are that it is only functional when the AT is already in an Active data call and crosses a subnet boundary. If the call continues long enough such that the AT requests a pilot from a DOM in the adjacent subnet that is not secondarily homed to the source or anchor RNC then the call will drop because the Resource Lookup on the anchor RNC will fail. Implementation of the inter RNC active soft handoff feature requires the secondary sector border to be planned such that coverage between the primary sectors and the secondary sectors is contiguous to enable soft handoffs to occur. To prevent dropped connections the secondary sectors should be in areas of low mobility such as secondary roadways or through areas where there are not high concentrations of fixed


users. Care should be taken in border planning such that there will be a dominant PN in the coverage overlap region between border sectors. The absence of a dominant server in these areas may lead to Ping Pong. Excess ping pong can lead to excessive session setups, access failures and dropped connections. The following figure illustrates secondary sector deployment in an area of high mobility. The secondary border can be defined as Subnet 1+ in the figure below. The shape of the secondary border for Subnet 1 will allow AT’s originating calls in Subnet- 1 to move to sectors deep within Subnet-3 and Subnet -7 without dropping the active call and would be ideal for extending coverage in Subnet-1 along the highway or any other high speed road [9].

(Subnet-3)RNC-3

RNC-4(Subnet-4)

RNC-5(Subnet-5)

RNC-6(Subnet-6)

(Subnet-7)RNC-7

(Subnet-2)RNC-2

RNC-1(Subnet-1)

Subnet-1+Highway or otherhigh speed road

Figure: IRAHO Example Showing the Ability to Maintain an Active Connection from RNC-7 -> RNC-1 -> RNC-3

The level of mobility can be seen as the percentage of calls entering or leaving a cell at any particular period of time. It could be expressed by the following ratio:

Cell Coverage

ATs entering cell ATs leaving cell

A C

BATs staying in cell

Cell Coverage

ATs entering cell ATs leaving cell

A C

BATs staying in cell


Level of mobility = (A + C) / (A + B + C) One way to get an idea of low mobility areas would be to look at the 3G data mobility behavior of the underlying 1xRTT network. There is no formula that can provide an exact mobility ratio, but the BTS OMs FCHOriginationNonBlocking3GData and the FCHHandoffNonBlocking3GData can help to provide an educated guess. They can be used in the following ratio which is similar to the mobility ratio above: Level of mobility ≈ (FCHHandoffNonBlocking3GData) / (FCHHandoffNonBlocking3GData + FCHOriginationNonBlocking3GData) Similarly, one way to get an idea of the level of data activity in a 1xEV-DO network is to analyze the data activity in the underlying 1xRTT network. This can be done by comparing, on a per sector basis, the data Minutes of Use (MOU) within the 1xRTT network. On a Nortel Networks 1xRTT data network, the MOU can be determined using the 3G Data MOU, i.e. PrimaryFrameCntFCH[3,5,7] from the BTS performance OMs. Lower amount of 3G Data MOU would indicate a lower data activity area. After initial deployment, RNC logs should be collected and analyzed using the NEWTUN tool. The NEWTUN tool will provide a list of DOMs that should be homed as secondary DOMs to the RNC under consideration. The use of NEWTUN to refine the DOM homing configuration will be iterative but should be highly refined after about the 3rd round of analysis. Setting and managing the number of primary and secondary DOMs per RNSM is done at the DO RNC’s topology manager. In addition to managing the number of primary and secondary DOMs per RNSM, each DOM has a primary selection table used for data filling the IP address of the primary RNC. Likewise there is a secondary RNC homing table which contains fields for adding the IP addresses of secondary RNC’s and a field the indicates whether secondary homing has been enabled or not [9].

Example DOM Menu Config Primary / Candidate and Secondary RNCs

3.1.5 CIQ Review and Provisioning Verification Prior to the 1xEV-DO RF optimization, it is a good practice to review the 1xEV-DO CIQ that the customer filled out and check that the 1xEV-DO network design matches the requirements listed in the CIQ response. Several important lists/information to check are as follows: • IP address design for RNCs, nodal elements, and DOMs • Inter Subnet borders and assignment of secondary RNCs to border sites – up to 8 secondary RNCs

homed per sector.


• Unique Color Code value for each RNC • EVDO Neighbor List – maximum of 20 neighbors per sector • Traffic Engineering – data application call model, distribution of services, and subscriber counts Reference [3] provides the complete provisioning rules and procedures for the DOM, DO-RNC, DO-EMS, and the backhaul networks. Of particular interest to RF engineers is probably the DOM provisioning. Currently each Metrocell can house three DOMs. However, there can be only one DOM per carrier (for 3 sectors). Therefore, if 1xEV-DO is deployed in only one carrier, the maximum number of DOMs is the same as the number of Metrocells. If the required number of DOMs is bigger than the number of Metrocells, then cell splitting may be considered. The required number of DOMs depends on the DOM capacity limitation, reverse link Erlang requirement, forward link throughput requirement, and the Metrocell limitation mentioned above [2]. A DOM has 96 channel elements and can serve about 90 connections in the reverse link (across 3 sectors). Assuming an average CE/user of 1.5, the number of primary users that can be served by a sector in a tri-sector site is 20 [1].


4 1xEV-DO Optimization Events

4.1 Initial RF Parameters Verification Since 1xEV-DO introduces many new RF parameters, it is important to verify that the correct RF parameters have been configured. The default RF datafill parameters are the recommended starting point for optimizing a network. Please refer to [3] and [8] to get detailed descriptions these parameters and their recommended values. EVDO Editor is a PC based application for viewing and editing specific RF parameters at the DOM level (i.e. RF Gain, Cell Radius, RAB Offsets, and Neighborlists). It also will allow users to view parameters at the RNC level. Currently, EVDO Editor has business rules checking capabilities for DOM level parameters. Future releases will have business rules checking capabilities for RNC level parameters. EVDO Editor can be found at http://navigate.us.nortel.com/imds?pg=/eng/wne/core/tool/adm/cdma/doed. EVDO Editor

4.2 Pre Optimization System Health Check The basic system readiness checkup is done to verify that the 1xEV-DO system is ready to carry traffic. In summary, the following attributes should be checked: • Verify status of all Nodes (DOM and RNC) using “NORTEL-XX# show node” have an administrative

and operational status of “Up” • Verify status of all Modules (BIO Cards, RNSM, SC, DOM, etc) using “NORTEL-XX# show module”

has a status of “Active”. • Verify status of all the interfaces (i.e. Ethernet, T1/E1, PPP, node 1/0/1) using “NORTEL-07# show

interface” command. All interfaces must have an administrative and operational status of “UP”.


o Ethernet interfaces: DOM Backhual – If a T1 is connected to a DOM along with 100/10 base T ethernt

connection, the DOM will prompt traffic from the RNC over T1. 2 Ethernet ports per BIO card – for support of A10/A11 interface communication,

Abis, A12, and OA&M paclets. On CLI, BIO Ethernet interfaces are listed as Ethernet 1/x/1, 1/x/2 where x denotes the slot of the card.

1 Ethernet port per SC control module - these are optional interfaces for carrying OAM traffic. On CLI, the active SC port is listed as Ethernet 1/0/1.

Management IP Interface – an optional interface that is used in conjunction with the SC aux. Ethernet if operators want to forward management IP traffic to a virtual management interface. On CLI the management IP interface is listed as mgmt1/0/1.

o All Used T1/E1: DOM Backhaul – used to backhaul DOM physical and link layer traffic to the

aggregation router. o PPP Interface for each T1/E1 interface

DOM Backhaul - a PPP link is associated with each T1 between the DOM and the aggregation router to provide duplexed, simultaneous, bidirectional, sequential, packet transfer of encapsulated Abis and OAM IP packets between two dedicated network peers. On CLI, the DOM T1 / PPP interface is denoted as: ppp1/0/1, ppp1/0/2, ppp1/0/3, and ppp1/0/4 interfaces.

o Node 1/0/1 interface Abis and OAM data - provides a loop back or virtual interface for the transfer of

Abis data from the DOM to the primary RNC and OAM data to the EMS. On CLI it is shown as : node 1/0/1 interface.

• Status of the Abis links between the DOM and the RNC using “NORTEL-07# show abis peer”. Every RNC should have an Abis status of “Connected” to every DOM that is homed to it.

• Status of the traffic and control channels established between the DOM and the RNC. • Ping the Node IP address of DO RNC, IP Addresses on the aggregation router associated with DOM

backhaul links, and the DO EMS IP address from the DOM. “NORTEL-07> ping 10.0.0.0” • Ping the IP address of all DOMs, IP addresses of DOM PPP Links, all PDSNs, and the DO EMS from

the RNC. • Status of GPS for all DOMs. “NORTEL-07> show GPS health” The basic system readiness checkup should be done from the EMS. An example of how to do it from the command line interface can be found in Appendix B. In the event any of the system readiness checks fail refer to the troubleshooting or recovery sections of reference [16].


4.3 FTAP and RTAP Testing FTAP and RTAP tests should be performed to network end to end performance capabilities inclusive of the EVDO airlink. FTAP and RTAP are built in test EVDO testing utilities that send test frames to determine the available bandwidth of the network. Prior to testing, a location where RF is known to be good should be selected. In addition, AT logs should be captured during TAP testing to measure the PER% seen at the AT.

Data rates given by FTAP tests should be equivalent to the average DRC request rate and RTAP tests should show equivalence to the average Reverse link Rate limit of the sector.

The following is an example of how to configure TAP tests from CLI: For more information on TAP tests see, reference [15].


4.4 TCP / IP Parameters Prior to starting optimization a review of configured TCP / IP parameters should be performed. Due to the fact the FTP protocol, used extensively in optimization activities, is based on TCP, a thorough review of configurable TCP and PPP layer parameters should be performed to insure accurate performance reporting. See Reference [13] for more information on TCP/IP parameter descriptions. Recommended TCP/IP Configuration:

• Data Collection Laptop configuration – Insure that the following TCP / IP settings are optimized. TCP send and receive buffers should be optimized according to the bandwidth delay product (i.e. Buffer Size = Bandwidth x delay). For EVDO, ideal Buffer Size = (2450kbps/8bits/byte * 150msecs) = 45.9kBytes.

o Recommended TCP RX Window Size = 64240

TcpWindowSize (Configured from Registry Editor)

Key: Tcpip\Parameters, Tcpip\Parameters\Interface\interface

Value Type: REG_DWORD—number of bytes

Valid Range: 0–0x3FFFFFFF (1073741823 decimal). In practice the TCP/IP stack will

round the number set to the nearest multiple of maximum segment size (MSS). Values

greater than 64 KB can be achieved only when connecting to other systems that support

RFC 1323 Window Scaling.

Description: This parameter determines the maximum TCP receive window size

offered. The receive window specifies the number of bytes that a sender can transmit

without receiving an acknowledgment. In general, larger receive windows improve

performance over high-delay, high-bandwidth networks. For greatest efficiency, the

receive window should be an even multiple of the TCP Maximum Segment Size (MSS).

This parameter is both a per-interface parameter and a global parameter, depending

upon where the registry key is located. If there is a value for a specific interface, that

value overrides the system-wide value. See also GobalMaxTcpWindowSize.

o Recommended MTU Size = 1500

MTU (Configured from Registry Editor)

Key: Tcpip\Parameters\Interfaces\interface

Value Type: REG_DWORD—number

Valid Range: 88–the MTU of the underlying network

Default: 0xFFFFFFFF

Description: This parameter overrides the default Maximum Transmission Unit (MTU)

for a network interface. The MTU is the maximum packet size, in bytes, that the

transport can transmit over the underlying network. The size includes the transport

header. An IP datagram can span multiple packets. Values larger than the default for the

underlying network cause the transport to use the network default MTU. Values smaller

than 88 cause the transport to use an MTU of 88.

• Note: Windows 2000 TCP/IP uses PMTU detection by default and queries the NIC

driver to find out what local MTU is supported. Altering the MTU parameter is

generally not necessary and may result in reduced performance.

• PMTU Discovery (Enabled)


When a connection is established, the two hosts involved exchange their TCP maximum segment size (MSS) values. The smaller of the two MSS values is used for the connection. Historically, the MSS for a host has been the MTU at the link layer minus 40 bytes for the IP and TCP headers. However, support for additional TCP options, such as time stamps, has increased the typical TCP+IP header to 52 or more bytes.

EnablePMTUDiscovery

Key: Tcpip\Parameters

Value Type: REG_DWORD—Boolean

Valid Range: 0, 1 (false, true)

Default: 1 (true)

• IP Header Compression (Disabled) It is well known (and has been shown with experimental data) that TCP header compression does not perform well in the presence of packet losses. If a wireless link error is not recovered, it will cause TCP segment loss between the compressor and decompressor, and then header compression does not allow TCP to take advantage of Fast Retransmit Fast Recovery mechanism. The header compression algorithm does not transmit the entire TCP/IP headers, but only the changes in the headers of consecutive segments. Therefore, loss of a single TCP segment on the link causes the transmitting and receiving TCP sequence numbers to fall out of synchronization. Hence, when a TCP segment is lost after the compressor, the decompressor will generate false TCP headers. Consequently, the TCP receiver will discard all remaining packets in the current window because of a checksum error. This continues until the compressor receives the first retransmission which is forwarded uncompressed to synchronize the decompressor. As previously recommended, header compression SHOULD NOT be enabled unless the packet loss probability between the compressor and decompressor is very low. Actually, enabling the Timestamps Option effectively accomplishes the same thing. Other header compression schemes like RFC 2507 and Robust Header Compression are meant to address deficiencies in RFC 1144 header compression. Figure 4 – Disabling IP Header Compression in Windows 2000 In order to disable IP header compression, go to Start -> Settings->Network and Dial Up Connections, and then select the appropriate device and then “Properties”. After selecting properties, then go to the TCP/IP component under the Networking tab and select properties again.


• Software data compression (Disabled) Data compression enables information to be transmitted beyond the actual connection speed. Data, particularly text and graphics, usually contain repeated sequences of identical information. Data compression works by replacing many characters of repeated information with a few characters and transmitting only one copy of repeated sequences of data. Communication software, such as Network and Dial-up Connections, may support data compression. For example, using a 14.4 Kbps V.32bis modem, you can enable software compression, and experience an average throughput of 28.8 Kbps. Tests show that software compression can result in higher data transfer rates than hardware compression. To insure accurate reporting of application or physical layer throughput over the access network the “Enable Software Compression” setting in the Windows “Dialup Networking Properties - >> PPP Settings” should be disabled. Figure – Disable Software Compression


• LCP Extensions (Disabled)

LCP is part of the PPP protocol that provides for the establishment, configuration, and testing of the peer to peer data link connection. The enable LCP extensions option within the PPP settings of Windows Dial Up Networking allows the software to take advantage of additional LCP features. Some extensions for LCP and there functions are as follows and more detail provided in http://ietfreport.isoc.org/idref/rfc1570/#ref-2 1. Identification [12] – allows the application to identify itself to its peer (Link Maintenance

Packet) 2. Time Remaining [13] – notifies the peer of the time remaining in the session 3. Frame Check Sequence (FCS) Alternatives [9] – provides a method to specify another FCS

format to be sent by the peer or to negotiate it away altogether. 4. Self Describing Padding [10] – Indicates to the peer that the use of padding is understood.

Used when some protocols, such as network layer or compression protocols, are configured which require detection and removal of any trailing padding.

5. Callback [13] – provides a method for the implementation to request a dial up peer to call back.

6. Compound Frames [15] – allows the implementation to send multiple PPP encapsulated packets within the same frame. (self describing padding must be used in conjunction with this option)

• TCP Timestamps (Enabled)

Another RFC 1323 feature introduced in Windows 2000 is support for TCP time stamps. Like SACK, time stamps are important for connections using large window sizes. Time stamps were conceived to assist TCP in accurately measuring round-trip time (RTT) to adjust retransmission time-outs. The use of time stamps is disabled by default. It can be enabled using theTcp1323Opts registry parameter.

Tcp1323Opts

Key: Tcpip\Parameters

Value Type: REG_DWORD—number (flags)

Valid Range: 0, 1, 2, 3

0 (disable RFC 1323 options) 1 (window scale enabled only) 2 (timestamps enabled only) 3 (both options enabled) Default: No value; the default behavior is as follows: do not initiate options but if requested

provide them.

Description: This parameter controls RFC 1323 time stamps and window-scaling options. Time

stamps and window scaling are enabled by default, but can be manipulated with flag bits. Bit 0

controls window scaling, and bit 1 controls time stamps.

• Selective Acknowledgment (SACK) - (Enabled)

SACK is especially important for connections using large TCP window sizes. Prior to SACK, a receiver could only acknowledge the latest sequence number of contiguous data that had been received, or the left edge of the receive window. When SACK is enabled, the receiver continues to use the ACK number to acknowledge the left edge of the receive window, but it can also acknowledge other non-contiguous blocks of received data individually. When SACK is enabled (the default), a packet or series of packets can be dropped, and the receiver can inform the sender of exactly which data has been received, and where the holes in the data are. The sender can then selectively retransmit the missing data without needing to retransmit blocks of data that have already been received successfully.


SACK is controlled by the SackOpts registry parameter. The Network Monitor capture below illustrates a host acknowledging all data up to sequence number 54857341, plus the data from sequence number 54858789-54861685. SACK is enabled by default in Windows

+ FRAME: Base frame properties + ETHERNET: ETYPE = 0x0800 : Protocol = IP: DOD Internet Protocol + IP: ID = 0x1A0D; Proto = TCP; Len: 64 TCP: .A...., len:0, seq:925104-925104, ack:54857341, win:32722, src:1242 dst:139 TCP: Source Port = 0x04DA TCP: Destination Port = NETBIOS Session Service TCP: Sequence Number = 925104 (0xE1DB0) TCP: Acknowledgement Number = 54857341 (0x3450E7D) TCP: Data Offset = 44 (0x2C) TCP: Reserved = 0 (0x0000) + TCP: Flags = 0x10 : .A.... TCP: Window = 32722 (0x7FD2) TCP: Checksum = 0x4A72 TCP: Urgent Pointer = 0 (0x0) TCP: Options TCP: Option Nop = 1 (0x1) TCP: Option Nop = 1 (0x1) + TCP: Timestamps Option TCP: Option Nop = 1 (0x1) TCP: Option Nop = 1 (0x1) TCP: SACK Option TCP: Option Type = 0x05 TCP: Option Length = 10 (0xA) TCP: Left Edge of Block = 54858789 (0x3451425) TCP: Right Edge of Block = 54861685 (0x3451F75)

• FTP Server (Sun) XMIT Hi Water mark

o If the FTP server is a SUN Solaris workstation, insure that the TCP XMIT HIWAT setting is configured for 64k.

o Windows based FTP servers have no configurable TCP TX buffer setting. o ndd -set /dev/tcp tcp_xmit_hiwat 65536

• The tcp_xmit_hiwat parameter is effectively the size of the send buffer. The send

buffer high watermark tunes the transport layer socket buffer size on a kernel wide

basis. The socket buffer can also be changed on a per-socket basis by using the

SO_SNDBUF socket option within an application. Mind that for UDP the size of the

output buffer represents the maximum datagram size.

4.5 Shakedowns The first step in 1xEV-DO RF optimization is to perform a shakedown of each cell site and all inter subnet borders. Like in CDMA, the objective of the shakedown is to verify that we can get a connection on each sector and switch to the adjacent sector in the forward link (and softer-handoff to it in the reverse link), and to ensure that the sector RF parameters (like PN offset and neighbor list) are correct. Chapter 6 presents the detailed procedure to do the shakedown.

4.6 Golden Value (Stationary) Testing The second step in 1xEV-DO optimization is to perform a stationary data testing to measure the maximum single-user application throughput in the forward and reverse link. During this exercise, the TCP, RLP, and/or other RF parameter setting can be optimized to improve the throughput. Once those settings are optimized and finalized, the measured single user application throughput becomes the “golden” value, i.e.,


the upper bound of what we can achieve in the network. Chapter 7 presents the detailed processes involved in performing stationary (Golden Value) data testing.

4.7 Cluster Drive Testing Once the stationary data testing is done, we can do drive tests along the important roads to measure various RF performance in a cluster. We can also do more stationary data testing in different locations (that are deemed important) throughout the cluster. Based on these test results, the datafill (i.e. Power and resources related parameters, etc) can be adjusted to improve the 1xEV-DO performance. Chapter 9 details the process involved in performing typical 1xEV-DO cluster drive testing.

4.8 Troubleshooting If RF performance problems are encountered during the stationary and drive testing (or anytime during the deployment period), engineers will have to analyze various performance attributes in efforts to diagnose where the root cause lies. Depending on the problems, troubleshooting may require more than the regular mobile logs obtained from the cluster drive testing. Chapter 10 presents various tips and examples to help engineers troubleshoot various 1xEV-DO RF performance problems.


5 Data Collection / Post Processing Tool Setup

This chapter will discuss briefly the requirement and the setup procedure for 1xEV-DO RF data collection and post-processing tools. More detailed discussion can be found in the reference documents mentioned in each section.

5.1 Laptop Setup The preferred method of 1xEV-DO data collection/testing is using a laptop connected to WirelessLogix Optis-M tool (EV-DO version) or Andrew Invex3G. If only one AT is used (like in the shakedown), then XCAL-DO or Invex3G PC can be used.

Laptop connected to Optis-M data collection tool.

Laptop connected to Invex3G data collection tool

The laptop used for 1xEV-DO testing has to have the following: • Windows 2000 or XP operating system • Intel Pentium4 processor 1.6 GHz equivalent or above • 512MB RAM or above Please refer to [12] for more detailed specification requirements. The following software needs to be installed in the laptop:


Drive Test Data Collection Tools: • WirelessLogix Optis-M / XCAL -DO version 2.2.0 or later.

o Available from Couei website: http://pms.couei.co.jp o Login with user id = [email protected] and password = WLsupportNortel

• Andrew Invex3G / PC version 4.3 or later. o http://www.andrew.com/products/measurement_sys/Invex/

Drive Test log Post Processing Tools: • EV-DO RF Optimizer version 2.2.5.30 or later.

o Available at http://navigate.us.nortel.com/imds?pg=/eng/wne/core/tool/cdma/rfoevdo) • XCAP-DO ((from WirelessLogix) version 3.77 or later can also be used.

o Available from Couei website: http://pms.couei.co.jp o Login with user id = [email protected] and password = WLsupportNortel

DOM-RNC logs post-processing tools: • NEWTUN Available at: http://navigate.us.nortel.com/imds?pg=/eng/wne/core/tool/adm/cdma/ntn • LogFileConvert (Java tool) (Available on RNC) • LogConvertDOS (Available on RNC) Applications: • FTP Client application such as WS-FTP Pro. Chariot may also be used. Optional tools: • TCP performance logging tools: Windump, Snoop, Ethereal and TCPTrace. • OM analysis tool: OMAX (Nortel OM analysis tool available through Mike Anderson) • There are several TCP parameters that need to be set correctly. See Section 4.4 above. If either Optis-M or Invex3G are used as the data collection tool, then the TCP parameters need to be set inside the tool. However, if either XCAL-DO or Invex3G - PC are to be used, then we need to change these parameters in the laptop. One easy way to do it is by using DrTCP utility software (available in the internet, for example from http://www.dslreports.com/drtcp). Section Appendix 4.4 contains a detailed description of settable TCP / IP paremeters on how to optimize them for the data collection tools (i.e. laptop, etc) and the FTP server.

5.2 AT Setup The ATs (mobiles) that are recommended to be used for 1xEV-DO RF optimization along with data collection platform support are given below: • Qualcomm QTP-6500 (ZRF 6500) – Optis / XCAL / Invex3G • Sierra Wireless 580 – Optis / XCAL / Invex3G • Sierra Wireless /Audiovox 5220 – Optis / XACL / Invex3G • Samsung A890 – Optis / XCAL • LG Vx8000 – Optis / XCAL • Novatel Merlin V620 – Optis / XCAL / Invex3G If XCAL-DO or Invex3G - PC is used as the data collection tool then the right driver for the AT needs to be installed in the laptop.

5.3 Data Collection Tool Setup

5.3.1 RNC Logging Setup RNC logs should be collected for the following tests:


• A13 Dormant Session Handoff Shakedowns • Cluster Drive Tests Note: The timestamps given in RNC logs are based on GMT time. In order to correlate the RNC logs with AT logs it will be necessary to adjust the RNC log timestamps according to the difference in local time and GMT time. For a fully loaded DO-RNC, there is one active System Controller on slot 7, one redundant SC on slot 9, one hot swap SC on slot 10 and one hot swap redundant SC on slot 8. A single DO-RNC can have up to four RNC-BIO modules and up to eight RNC-RNSM modules. For a DOM, there is one BIO/SC module and one 1xEVDO modem. The 1xEVDO-Modem contains two modules, Forward Link Processor Module (FLM) and Reverse Link Processor Module (RLM). Given that call processing resources could be assigned to any cards in the RNC, logging has to be enabled on all the cards (RNSM or SC) in the system. Logging can be configured through the DO EMS or CLI to configure the logging system of each individual card. Logging Trap Severity Within each hardware module that has a CPU, various software components generate all kinds of events. The verbal description of these events is considered as “event messages”. Those “event messages” can be selectively captured as a “log” by logging system of the module. The selective capturing functionality is implemented by setting up a set of input traps which is also called “Severity”.

So in the end, the captured “logging messages” by the logging system are the subset of all event messages generated by those software components within the hardware module. The EVDO system supports severity levels from 1-32. For most events a severity level of 8 is all that is required. Other events like “Forward Sector Switching” may require logging severity levels of 22.

The following is shows the components of an RNC log message:

Configure the Log File Both the DO-RNC have there own logging manager where the logging manager is the means of configuring the properties of the Log File. The log file itself is the actual captured log saved on the RNC SC, RNSM or the DOM BIO/ SC cards.

Time Stamp of the Date

CPU IDMessage ID

6-10-04 10:19:47.241 S=16 C=010301 F=0009 ID=0397 [0x3403ef (uati) SCSM SCSM_Open] : Received Connection Opened Indication event

Message Severity Level

Time Stamp of hh:mm:ss.sss

Component ID

Message Body

Time Stamp of the Date

CPU IDMessage ID

6-10-04 10:19:47.241 S=16 C=010301 F=0009 ID=0397 [0x3403ef (uati) SCSM SCSM_Open] : Received Connection Opened Indication event

Message Severity Level

Time Stamp of hh:mm:ss.sss

Component ID

Message Body


• To configure the log files using the DO-EMS go to “Network Database” “DO-RNC” RNC Menu Show Cards as shown below.

• Select the SC Card from the Cards Menu • From the SC Card Menu select Show logfacilityMgr Modify Logfacility 5 (Call Control) • The following boxes need to be configured in the Call Control Log facility Manager on the SC Card

o Logging Enabled = True o Maximum Severity = 5, 8, 16, 22, 32, etc o Output to File = On o Maximum Severity on File = Maximum Severity entered above

DO-EMS RNC Menu


RNC Cards Menu

SC Card Menu

Log Facility Manager


Call Control Log Facility

Once all changes are made to the Call Control Log Facility Manager select Submit

To perform a severity level range selection, for example from 1 to 16 on call-control component do the following from the CLI prompt:

NORTEL-07>en NORTEL-07#config NORTEL-07(config)#logging trap severity 16 call-control To perform a set of individual severity levels selection, for example severity level 5, 9 and 16 on call-control component. NORTEL-07>en NORTEL-07#config NORTEL-07(config)#logging trap severity fatal call-control NORTEL-07(config)#logging trap severity 5 call-control + NORTEL-07(config)#logging trap severity 9 call-control + NORTEL-07(config)#logging trap severity 16 call-control + NORTEL-07(config)#logging trap severity fatal call-control - NOTE: To configure a set of individual severity level selection, users need to first specify a range of severity setting then add each individual severity level one at a time with “+”. Finally, users need to remove the first range severity setting with “-” sign.

To access those non-SC modules, users need first login into those SC modules. From there use following procedure to access and configure the input traps on those modules.


NORTEL-07>en NORTEL-07#config NORTEL-07(config)#module-logging bio1/11/1 NORTEL-07(module-logging-bio1/11/1)#trap severity 16 resource-control NORTEL-07>en NORTEL-07#config NORTEL-07(config)#module-logging rnsm1/13/1 NORTEL-07(module-logging-rnsm1/13/1)#trap severity 16 call-control

To Enable or disable logging from the CLI prompt do the following:

On RNC SC card for call-control component: NORTEL-07>en NORTEL-07#config NORTEL-07(config)#logging start call-control (To enable log capture) NORTEL-07(config)#no logging start call-control (To disable log capture)

Parsing and Retrieving Log Files To parse the logs under RNC: (This step is not necessary if using NEWTUN)

NORTEL-07(config)#logging convert file <rnc_binary_log.bin> <rnc_output_file.txt> No wild card supported under command line, so log need to be parsed one by one. To check the log files under RNC: NORTRL-07>shell

NORTRL-07(shell)(disk0)>cd logs NORTRL-07(shell)(disk0)>ls (To view all the files under the logs directory)

To get the log the log files, users can use various FTP programs to access the log directory and then extract those logs. Once those logs are download to users’ local hard drive, they can be viewed with any kind of text file editor program. For more detailed information on RNC logging see reference [4].

5.3.2 Optis-M Setup Once the Optis-M GUI is installed, the user needs to do the following: • Connect AC/DC Converter’s input connector to power source. • Connect AC/DC Converter’s output connector to the Power supply connector on the front panel of

Optis-M. • Connect Ethernet cable to Ethernet connector on the front panel of Optis-M and laptop. • Turn on the Optis-M unit using the power switch on the front panel. • Connect Mobile data cable to the USB port of each slot on the front of Optis-M. • Connect GPS Antenna to the GPS Antenna connector on the front panel of Optis-M. • Set the laptop IP address to the appropriate one (usually 1.1.1.1) so that it can communicate with

Optis-M box. • Start the Optis-M software. Ensure that the client hardware status light (at the bottom right corner of

Optis-M window) is green. • Configure the Optis-M ports. Below is an example for two Chesters connected to Optis-M.


Port Setting window in OPTis-M.

Phone Type setting under Port Settings window.


Data Port Configuration under the Port Settings window • Set the log masks. For more details on setting up Optis-M, please refer to [10].


5.3.3 Optis-M Log Masks The log mask settings for the general 1xEV-DO data collection (e.g. cluster drive testing) are shown in the pictures below. The 1xEV-DO log masks are always needed. The CDMA/1xRTT log mask is needed for measurement with hybrid AT (e.g. for 1xEV-DO to 1xRTT handoff analysis). For a specific testing or troubleshooting, different log masks may be needed.


OPTis-M 1xEV-DO log mask.


OPTis-M CDMA/1xRTT log mask setting.


5.3.4 Invex3G Chassis Setup • Note: Invex3G Chassis, Release 4.3 / 4.4 should be provisioned with at least 2 DI (digital

communications interface) boards for EVDO optimization testing. The legacy CI boards are processor limited which will have negative consequences when measuring data throughput performance.

• Note: If Invex3G DI boards are not available at the commencement of optimization testing,

Invex3GPC or Wirelesslogix tools should be used to insure success. Invex3G Chassis - Setup • Connect the RJ45 Ethernet crossover cable from the System Controller Module (GWSC-0100) of the

Invex3G mainframe to the Network Interface Card of the Personal Computer (laptop) that will be used to run the Invex3G software.

• Connect the GPS antenna to the GPS ANT connector on the mainframe. • Insert the AT (PCMCIA) into the Digital Card Interface (DCI) Module or attach the AT to the

Communications Interface Card (CI) using the appropriate connector cable to the CI ports (STS-A and STS-B).

Invex3G mainframe also charges all ATs that use the proprietary cable interface to the CI board. The upper port (A) of the CI module supports Data and Voice application, and the lower port (B) is

reserved for Voice ONLY. Therefore if doing a data call, use ONLY the upper ports in the Communication Interface (CI) Card.

Each CI port and DI card slot has dedicated LED Indicators (STS-A and STS-B) for status check: • Green: Phone Attached - Normal • Red: Fault • Amber: Waiting • Connect the power supply to the Invex3G unit.

DI Card Slot


• Open the start menu on the laptop and go to “My Network Places” or Network and Dial up Connections” and select the “Local Area Connections” icon and then select properties.

• On the “General” tab go the Internet Protocol (TCP / IP) component and select “Properties” • On the “General” tab select “Use the Following IP Address” and enter the following:

o IP Address – 192.168.3.100 o Subnet mask – 255.255.255.0

• Click OK and Close the Networking tab • Power the laptop and the Invex3G unit. The unit can be powered either by connecting directly to the

cigarette lighter or using an inverter in the van. • Once the software has been launched it is necessary to create new device connections if they were not

created beforehand. o Click on the Connections tab in the workspace

o Click on the Connections folder icon and select “New” connection

o Next, the “add new connection” dialog appears. There are three options available for

connection configuration: o Invex3G Chassis – specifies connection to the hardware mainframe o Invex3G Serial Device – specifies connection to a device attached to

the PC (Invex3G-PC option)

o Select Invex3G default chassis option and then OK. o Select “Connect All” from the workspace connections menu to open the connection to the

Invex3G chassis.

o After selecting connect all the “Open Connections” dialog begins. Click the Andrew

chassis name to highlight it and then click OK.


o The Invex connection status dialog box will open up to display the connection progress.

At the initial start up it may take several minutes for the chassis flash memory to update so, be patient.

• Once the connection process is complete a list of devices attached to the chassis will be displayed in the devices tree

5.3.5 InVex3G Log Masks Configure Invex3G logs masks as follows for EVDO and CDMA log collection.

• Go to the “Devices” tab in the Invex3G workspace and right click on the collection device and select properties. The following window will appear.


• Select the CDMA L3 and QCP 1xEVDO L3 tabs and click “Collect 1xEVDO Layer 3 Messages”

and “Collect CDMA Layer 3 Messages” boxes. • Select the QCP 1xEVDO tab and then select all messages. • Select the QCP1X tab and select all messages. • For more information on Invex3G configuration see reference [18].

5.4 AT Log Data Post-Processing Tool Setup

5.4.1 RF Optimizer EV-DO RF Optimizer EV-DO software can be downloaded from the link below: http://navigate.us.nortel.com/imds?pg=/eng/wne/core/tool/cdma/rfoevdo Then follow the installation procedure carefully. Once RFO is installed, attach a RFO dongle key and launch RFO. Go to Tools Options to configure the tool:


• In the "File Locations" tab, be sure to select a location for "Extraction Logs" that you can easily locate later. The extraction log accumulates many informative messages during raw processing. Since new messages always get appended to the existing file, the file can become quite large. It's a good idea to periodically delete the extraction log once you have reviewed its contents.

• When selecting locations for "Database Data Files", "Extraction Logs" or "Temporary Files", be sure to select only folders to which you have full access rights, and on the drive that has plenty of available space. These files, created by RF Optimizer EVDO during processing, can consume a significant amount of disk space.

• In the "Binning" tab, set temporal and spatial bin sizes (the default setting of 1s and 100m is a good

starting point). Keep in mind that the smaller the bin sizes, the longer it will take RF Optimizer EVDO to process the data, and the larger the resulting databases will be.

• In the "Processing Options" tab, it is strongly recommended to keep the "Delay..." option checked.

This will delay the Data Synchronization analysis for any log file, which can be very time consuming due to the amount of data processed, until the first time you have a need to open the DRC Information Viewer. Keeping this option checked will significantly save upfront processing time, compared to not checking this option.

• You are ready to process data. • Please keep in mind that by Microsoft's design, each of your database (MSDE) is limited to 2GB. Due

to the varying nature of raw data, it is not possible to predict how much database space each raw file


will take up after it is processed, so exercise caution and avoid processing too many files into one single database. You are free to create as many databases as your hard drive can hold.

5.4.2 XCAP-DO XCAP DO software also needs a dongle key to run. Once launched, click on Tools Options to configure the tool. There are six tabs in the Options window, and most of the default values are good for typical use. • General: contain settings for distance unit, message viewer option, sync option, etc. • Path: select the directories for the log data (raw data), model (processed data), and exported data.

• Map: contains settings for map display. • Map2: contains additional settings for map display (like shift offset). • Graph: contains settings for graph display. • CDF/PDF: contains options for CDF and PDF graph type, i.e. line or bar and 2D or 3D. To process a log file (XCAP Do calls it “creating a new model”), click on File New:


• Sampling Interval: As in RFO, it is typically set to 1000 ms. • Merge/Separate: If you select more than one file, and want to create one model for them (e.g. to get an

aggregate statistics from a cluster), then choose “Merge”. If you want to create a model for each file separately, then choose “Separate”.

• Model Name: If you merge several files, you can choose a name for the merged model. If you choose “Separate” then the model name is the same as the individual file name.

• Selective Parsing: If you want to process only certain messages, you can select them from here. For

example, if you are measuring RF coverage and not concerned about throughput, then you can uncheck PPP Data and Throughput Info box.


5.5 DOM - RNC Log Post Processing Tools Currently, RNC logs can be analyzed one of two ways; converting the logs to text and opening the resulting logs with Excel, or to use NEWTUN to post process and analyze the logs in binary format.

5.5.1 Converting RNC Logs from Binary to Text The binary to text file parser resides in the RNC SC and DOM card. To parse a DOM or RNC log, you need to telnet into the DOM or the RNC. The following steps should be followed once logged into the DOM or RNC to convert the binary data to text:

• Nortel-07> en

• Nortel-07# shell

• Nortel-07#(shell)(disk0:/)# cd logs

• Copy the filename of the log

• Nortel-07(config)# logging convert file <bin filename> <new text file name>

• After converting the file ftp it down to a PC for analysis.

5.5.2 NEWTUN (Post Processing and Analysis of RNC Logs) NEWTUN (Nortel Engineering Wireless Tuning Tool) is a Windows based tool developed by Nortel’s Wireless Tools group for processing and analyzing EVDO call statistics according to events captured in RNC logs. NEWTUN is currently only a Beta release and is only available to select groups. To obtain a license key please contact Tag Support. The processing of RNC logs with NEWTUN requires the input files from the RNC to be in the raw *.bin or *.gz format (format prior to text conversion). During post processing of the logs NEWTUN will convert the logs to a viewable format. NEWTUN will not read RNC logs that are in *.txt format.

NEWTUN supports the following functionality in the current release: • Call Performance statistics such as dropped connection and access failure rates, • Call flow analysis in various states • Call radius configuration tuner • In the future it will support Neighborlist tuning and integration with EVDO editor output files

NEWTUN Setup Guidelines When first starting NEWTUN, the user should ensure that the tool settings are configured as required. File Locations


• It is recommended that the User save the Database files in a location where there is a lot of free disk space available when processing large amounts of data.

• NEWTUN creates a number of temporary files during processing that may be large depending upon the amount of data processed. These temporary files are cleaned up upon completion of processing activities.

• To save browsing time, the user can setup default location of where the program should search for raw binary logs.

Process Filtering


Filter Messages • When checked the tool will only process Call Control messages • Certain “useless” CC messages will also be excluded

Skip Files Currently does nothing but will allow automatic removal of files with Severity levels lower than

required by call model analyzer.

Max Severity • When calculating fail / drop statistics it is important that the log files are collected at the

appropriate severity level. Should the level be too low the tool will warn the user accordingly. Database Creation Raw data must be processed and stored into a database for post-analysis. To create a new database:

1. Open DataManager from Main Window Toolbar

2. Click New Database Tool button on Data Manager


3. Enter Name of new database File Processing NEWTUN processes raw binary Airvana RNC log files. For space reasons, the tool will accept gzipped binary log files. To process Files

1. Create database for storing data 2. Highlight database to store processed data 3. Select files for processing 4. Add files to highlighted Database 5. Repeat steps 1 – 4 to batch up data as desired 6. Click Process Button


Once all files have been successfully processed, select the database of interest and click the Select button. By default, the statistics view will automatically be populated for post-analysis.

Notes • When selecting data, all files within a database are automatically selected • Files cannot be removed from a database. • Clicking the delete button will remove the highlighted database • Tool does NOT process RNC converted text files

Statistics Display The statistics view is the main apparatus for displaying performance results and navigating throughout the call messaging for debugging purposes.


The statistics displayed are based upon all the calls identified within the database selected from the data manager.

Connection types

Two types of connections are defined within NEWTUN:

Session – Connections assisting with the configuration of a session startup Data – Connections associated with the transfer of user data

How these connection types are determined is beyond the scope of this document. As mentioned in the Known Issues section, break-up of connection types is not accurate and is undergoing improvements.

Call Classification Types Name Description Call Attempts Total number of Connection Attempts Failed Access Connections that failed to acquire traffic channel Dropped Calls Connections that dropped once on traffic channel Border Dropped Calls

Dropped calls that may be close to a border RNC region

Good Calls Calls that closed normally Incomplete Access Calls that had no more messaging to indicate outcome of Access Attempt Incomplete Traffic Call that had no more messaging to indicate whether call would terminate

successfully or not Unknown Failures Calls that failed but had insufficient information to determine whether traffic was


acquired or not Unknown Calls that were found (End of calls) but no information to classify was available Access Failure Rate (%)

UnknownFailuresUnknownIncompleteAccessAttemptsAccessFailed

−−−*100

Drop Call Rate (%)

DropsBorderDropsGoodDropsBorderDrops

+++

*100

Determination of how the calls are classified is beyond the scope of this document. Notes

All the classification types mentioned above apply to both Session and Data Connection types. Where possible, call drops / failures are pegged against a DOM / PN. If no DOM / PN information is available the call will be pegged under Unknown DOM entry.

Debugging Calls To review the cause of a drop or failure the user must Double-Click on the calls of interest within the Statistics View Tip Double-click a cell containing connections of interest to investigate will bring up message view


Message View The message view will display all decoded messages for the UATI relating to the call selected by the user.

The green color coded messages highlight the connection of interest (Not exact but provides an approximation) Note

• The Route Update Detail message has been re-formatted to accommodate later NEWTUN tuning module development.

• This message resolves all the rows of a Route Update providing one line per PN


Active List When the user selects calls to analyze from the Statistics view, this pane will show the list of calls for examination.

Double-clicking on any of the calls will update the message view with the messaging relating to that connection.

Note • It is recommended to not select a Statistics View Cell with too many calls (100 or more) to

minimize time delays during subsequent analysis.


Peg Selection To choose which metrics to display within the statistics, click the edit tool button.

Select the metrics to show by checking / un-checking as required.

Saving Results Later versions of NEWTUN will generate reports of results. Until that time users can save results to Excel for custom report generation.

5.6 OM Data Collection Template Configuration Setup The Nortel 1xEVDO system is very flexible for collecting operational measurements. The system includes many pre configured data collection templates that allow users to collection measurements with various degrees of granularity. In addition the system allows users to build customized data collection templates. Creating a data collection configuration allows the user to tailor the metrics to meet specific needs. The following instructions explain how to create a data collection configuration on the DO-EMS.

• Log into the EMS client as a user with PerformanceAdminGroup privileges. • Open the statistics administration window


• Click Add Data Collection Configuration from the menu. The DO-EMS will open the Data

Collection Configuration Form

• Configure the data collection parameters as follows:

o Configuration Name – Name of the data collection configuration o Start Time - Used for OM synchronization. The Start Time parameter specifies the

starting time of the first OM sampling, in date-time format: mm/dd/yyyy HH:MM. If the Start Time has already passed when the DC Configuration is “pushed” to the NE, then the first sampling will be taken at the next sampling time (using Start Time as the reference).

o End Time - Used for OM synchronization.The End Time parameter specifies the ending time of the first OM sampling, in date-time format: mm/dd/yyyy HH:MM. The end time validation is at the hour granularity.


o Sample Rate - The period at which the network elements in the configuration read and store the performance statistics. Select a sampling rate within the range 1 - 3600 seconds. It is not recommended to select rates below 900 s because it may degrade network performance, result in data collection errors, or impact OM synch time.

o Push Interval - The collected data is pushed from the NE to EMS at this time frequency. Select a push interval within the range 60 - 7200 seconds. It is not recommended to use values less than 1800s because it may result in data collection errors.

o Template - Select a data collection template from the list. The template determines which object IDs are sampled at the specified sample rate. Click the link to browse all existing templates.

o Instance IDs – to collect data on a specific instance list the instance ID here. If it is desired to collect data on all instances leave this field blank. For table based data collection templates it is not recommended to collect data on all instances as this may degrade NE performance.

• Select the node group to collect date from and submit the new template. • Data collection will start and end at the times specified. • Data collection results can be viewed on a per node basis in DO-EMS by selecting Data Collection

Files from the statistics view. • Alternatively, engineers can FTP the results down from the EMS using an FTP client.

5.7 CELLDM Data Collection CELLDM or Cellular Diagnostics and monitoring logs provide a means to retrieve airlink performance information for each sector from the baseband module. CELLDM has the following components: • CELLDM – SC – runs on the BIO/SC • CELLDM – FL – runs on the FLM • CELLDM – RL – runs on the RLM The airlink performance statistics comprise per sector and per user statistics. The per sector statistics are part of the OMs but the per users stats can be determined from CELLDM logging CELLDM provides the user two interfaces, NE CLI and GUI DO-EMS to access the airlink performance statistics and perform diagnostic logging of the baseband module. There are three different configuration options for CELLDM. These options are given in the table below.


CELLDM Configuration Options

Statistics managed by CELLDM are sector based, user based, and broadcast channel based. These statistics are always collected by different components. The sector and user based statistics are of two types: traffic statistics and signal quality statistics. Traffic statistics can be divided in to cumulative statistics and throughput statistics. Cumulative statistics These statistics are the sum total of various traffic-related variables associated with a sector or connection. These statistics are monotonically increasing as a function of time. Examples are: • The total number of bytes transmitted for a connection. • The total number of bytes received from the AT for a connection. • The number of slots for which a specific DRC value was requested by the AT. Throughput statistics These statistics are the time-average of the cumulative statistics. Examples are: • Forward link sector throughput (the rate at which bytes are transmitted out of a specific sector). • Average requested DRC (the sum total of all the DRC values requested by an AT averaged over time). Collection of statistics An operator can choose to collect any of the statistics using the method of data collection. Data collection is a feature of the software release and provides the ability to sample any MIB OID at a configurable rate. The CELLDM component also provides a mechanism that allows the operator to sample the statistics associated with a sector, connection or broadcast channel at a configurable interval and log these statistics to disk. • Sector-based traffic statistics • Sector-based signal-quality statistics • User-based traffic statistics • Broadcast channel statistics • User-based signal-quality statistics CELLDM Loggable Attributes The following operator provisionable attributes can be configured through the DO-EMS GUI or CLI. All attributes are configured on a per modem card basis


sectorStatsLoggingStatus Whether sector statistics are logged periodically to disk. sectorThruputStatsLogMask This mask should be set to log the sector throughput statistics to disk. sectorOverheadStatsLogMask This mask should be set to log the overhead (access and control)

channel statistics to disk. sectorBroadcastStatsLogMask This mask should be set to log the broadcast channel statistics to disk. sectorTCStatsLogMask This mask should be set to log the sector-based traffic-channel stats to

disk. sectorDrcChannelStatsLogMask This mask should be set to log the sector-based DRC channel stats to

disk. sectorRxQualityStatsLogMask This mask should be set to log the sector-based signal quality stats to

disk. connectionStatsLoggingStatus Whether sector statistics are logged periodically to disk. connectionThruputStatsLogMask This mask has to be set to enable the connection throughput statistics

to be logged to disk. connectionDrcStatsLogMask This mask has to be set to enable the connection’s DRC statistics to be

logged to disk. connectionTrafficStatsLogMask This mask has to be set to enable the traffic channel packet count

statistics to be logged to disk. connectionFlowStatsLogMask This mask has to be set to enable the connections flow statistics to be

logged to disk. connectionMiscStatsLogMask This mask has to be set to enable some miscellaneous ( ACK and PRC

channel) connection statistics to be logged to disk. broadcastStatsLoggingStatus Whether broadcast statistics are logged periodically to disk. broadcastGeneralStatsLogMask This mask has to be set to enable the general packet-count stats

associated with the broadcast channel to be logged to disk. statsSamplingInterval The period (in milliseconds) of how often the sector, connection and

broadcast statistics are logged to disk. throughputSamplingInterval The period (in milliseconds) of the duration of which instantaneous

throughputs are calculated. ftcSchedulerLogStatus Forward Traffic Channel Scheduler Logging status for this modem. cchSchedulerLogStatus Control Channel Scheduler Logging status for this modem. broadcastSchedulerLogStatus Broadcast Scheduler Logging status for this modem. AchLogStatus Access Channel Log status for this modem AchFingerLogMask Access Channel Finger Logging enable status for this modem. AchSearcherLogMask Access Channel Searcher Logging enable status for this modem. RtcLogStatus Reverse traffic channel logging enable status for this modem RtcFingerLogMask Reverse traffic channel finger logging enable status for this modem. RtcSearcherLogMask Reverse traffic channel searcher logging enable status for this modem. RtcHandoffLogMask Reverse traffic channel handoff logging enable status for this modem. RtcDopplerLogMask Reverse traffic channel doppler logging enable status for this modem. RtcDrcLogMask Reverse traffic DRC channel logging enable status for this modem RtcAckLogMask Reverse traffic ACK channel logging enable status for this modem RtcRriLogMask Reverse traffic RRI channel logging enable status for this modem RtcRpcLogMask Inner loop power control logging enable status for this modem csm5500DiagMask<Component> The Mask for controlling the diagnostics information of the specified

component for CSM5500-based baseband module units. A value of 1 means enable, 0 means disable.

csm5500VerboseMask<Component> Mask for controlling the verbosity of the diagnostics information of the specified component for CSM5500-based baseband module units. A value of 0 means terse, a value of 1 means verbose.

CELLDM Logging Setup 1. From the DO-EMS GUI, expand the Network Database DOM folders. 2. Select the DOM that where CELLDM logs are to be collected. 3. From the DOM Menu, select Config CELLDM.


4. There are three configurations for CELLDM logging available: CSM550CELLDMDiags, GlobalCELLDMLogging, and ModemCELLDMLogging.

CSM5500 CELLDM Diagnostics Configuration


CELLDM Global Logging Configuration

CELLDM Modem Logging Configuration


5. After the logging attributes have been configured select Submit to begin logging. Accessing Statistics In release 3.0, there is a new (Connections) pull down menu for accessing per user CELLDM statistics data 6. From the DOM Menu select Show Connections to access the per user statistics. 7. The available user Based statistics are as follows:

• FL Rate Statistics Information • RL Rate Statistics Information • Connection Sector Statistics Information • Flow Statistics Information • Flow Queue Statistics Information • Rx Signal Quality Information

8. In order to access per sector Statistics, from the DOM Menu select Config SectorElements. 9. Once on the Sector Elements page, go to the IS856 Sector Element Menu and select SectorPerfStats

10. The following Sector Performance stats can be viewed from this page.

• Access Channel Statistics Information • Broadcast Sector Statistics Information • Control Channel Statistics Information • FTC Information • RTC Information • Rx Signal Quality Information • General Sector Statistics Information

For More detailed information on CELLDM logging see reference [19].

5.8 OM Analysis Tools The core RF engineering team has developed OMAX – (OM Analysis for 1xEVDO and 1xRTT) for retrieving and calculating OM performance metrics for 1xEVDO and 1xRTT. The process of creating the OM statistics comprises; retrieving data from EMS, reading data into MySQL Control Center, and extracting performance data via SQL Queries. RAW OM data from EMS is organized into standard data collection templates (dc files). The data collected in the templates should be transferred to a local server where they can be FTP ed to a local PC with MySQL control center installed. Perl scripts are used on the PC analysis machine to parse the RAW OM data into a database (mySQL) MySQL can be used to run default performance queries or to build specialized queries for custom performance reports. The following is needed to install and configure MySQL:

• Files Needed o mysql-4.0.18-win.zipsly o mysql-4.1.1a-alpha-win.zip o mysqlcc-0.9.4-win32.zip

• The files can be found at the following locations: o http://dev.mysql.com/downloads/ o \\mikej-2\MySQLDownloads\rirs

• Unzip mysql-4.0.18-win.zip in temporary directory: o Run the Setup.exe o Follow setup instructions o Remove temporary directory contents when installation is complete


• Unzip mysql-4.1.1a-alpha-win.zip in temporary directory: o Copy the contents of the mysql-4.1.1a-alpha directory (including sub-directories) into

the installed mysql directory (i.e. c:\mysql) from previous step. o Remove temporary directory contents when complete

• Rarins un winmysqladmin.exe located in the mysql\bin directory o If prompted for user id and password, select ‘cancel’ o Should see a stop light (with green light) in the task bar o Server should be running in background and should automatically start after reboots.

• Unzip mysqlcc-0.9.4-win32.zip in temporary directory o Run Setup.exe o Follow setup instructions o Remove temporary directory contents when complete

• Obtain the following tar files: o mysql-4.1.1-alpha.tar o DBI-1.42.tarnalin

• DBD-mysql-2.9003.tara o \\mikej-2\MySQLDownloads\

• To compile MySQL client for Cygwin: o unpack the mysql-4.1.1-alpha.tar file o Don’t use windows to unpack the file; use tar -xvf o cd into the unpacked dir (i.e. cd mysql-4.1.1-alpha)

>./configure --prefix=/usr/local/mysql --without-server This prepares the Makefile with the installed Cygwin features. It takes some time, but should finish without error. The ‘prefix’, as given, installs the whole Cygwin/MySQL thingy into a location not normally in your PATH, so that you continue to use already installed Windows binaries. The --without-server parameter tells configure to only build the clients.

>make This builds all MySQL client parts … be patient. It should finish finally without any error.

>make install o This installs the compiled client files under /usr/local/mysql. o Remove unpacked dir

• To compile DBI for Cygwin: o unpack the DBI-1.42.tar file

>cd into the unpacked directory >make

>make test

>make install

o Remove unpacked dir • To compile DBD-mysql for Cygwin:

o Unpack the DBD –mysql-2.9003.tar file o cd into the unpacked dir

>perl Makefile.PL –-libs=“-L/usr/local/mysql/lib/mysql –lmysqlclient –lz” –-cflags=-I/usr/local/mysql/include/mysql –-testhost=127.0.0.1 >make


>make test # Some minor error message can be ignored here

>make install o Remove unpacked dir

• Documentation for MySQL can be found under the installed mysql/Docs directory (i.e. c:\mysql\Docs)

5.9 TCP Performance Logging Tools Windump and Snoop logs are generated to analyze the TCP performance between client (data collection computer – receiver) and server (FTP server – sender). Typically Windump would be installed on the data collection laptop connected to the AT and the Snoop logs would be installed on the FTP server such that the end to end TCP performance of the network can be analyzed during testing. Windump – Windump is a Windows portable version of “TCPdump” that runs on top of a low level packet capture utility called winpcap. Windump is freeware and can be downloaded from the following location: http://windump.polito.it/install/default.htm

Snoop – Snoop is a Unix command that is used to capture network packets in a readable format Snoop captures packets from the network and displays their contents. snoop uses both the network packet filter and streams buffer modules to provide efficient capture of packets from the network. Captured packets can be displayed as they are received, or saved to a file (which is RFC 1761- compliant) for later inspection.

Command Options:

-C List the code generated from the filter expression for either the kernel packet filter, or snoop's own filter. -D Display number of packets dropped during capture on the summary line. -N Create an IP address-to-name file from a capture file. This must be set together with the -i option that names a capture file. The address-to-name file has the same name as the capture file with .names appended. This file records the IP address to hostname mapping at the capture site and increases the portability of the capture file. Generate a .names file if the capture file is to be analyzed elsewhere. Packets are not displayed when this flag is used. -P Capture packets in non-promiscuous mode. Only broadcast, multicast, or packets addressed to the host machine will be seen.

-S Display size of the entire ethernet frame in bytes on the summary line.

Procedure:

1. Telnet into the server. 2. Login as administrator 3. Go to the directory where you want to save the log. 4. Type >/usr/sbin/snoop –o filename AT IP address <ENTER>

This will start the Snoop log. “o” here implies “Output to”

5. To stop and save the log “Ctrl C”


TCPTrace TCPTrace can take as input the files produced by several popular packet-capture programs, including tcpdump, snoop, etherpeek, HP Net Metrix, and WinDump. tcptrace can produce several different types of output containing information on each connection seen, such as elapsed time, bytes and segments sent and recieved, retransmissions, round trip times, window advertisements, throughput, and more. It can also produce a number of graphs for further analysis. Follow the link to download a copy of TCPTRACE: http://www.tcptrace.org/download.html

ETHEREAL Ethereal can read capture files from tcpdump (libpcap), NAI's Sniffer™ (compressed and uncompressed), Sniffer™ Pro, NetXray™, Sun snoop and atmsnoop, Shomiti/Finisar Surveyor, AIX's iptrace, Microsoft's Network Monitor, Novell's LANalyzer, RADCOM's WAN/LAN Analyzer, HP-UX nettl, i4btrace from the ISDN4BSD project, Cisco Secure IDS iplog, the pppd log (pppdump-format), the AG Group's/WildPacket's EtherPeek/TokenPeek/AiroPeek, or Visual Networks' Visual UpTime. It can also read traces made from Lucent/Ascend WAN routers and Toshiba ISDN routers, as well as the text output from VMS's TCPIPtrace utility and the DBS Etherwatch utility for VMS. Any of these files can be compressed with gzip and Ethereal will decompress them on the fly. Follow the link to download a copy of Ethereal: http://www.ethereal.com/download.html


6 Shakedowns

The first step of the 1xEV-DO RF optimization once RF datafill, FTAP / RTAP, and TCP/IP parameter verifications have all been performed is the shakedown of each BTS and RNC Border areas to verify call processing functionality and RF parameters/datafill.

6.1 Objectives The objectives of the shakedown are as follows: • To verify that we can get a connection on each sector. • To verify that we can switch to the adjacent sector in the forward link and softer-handoff to the

adjacent sector in the reverse link. • To verify that the following parameters have been datafilled correctly for each sector:

o PN offset o Neighbor list

• To verify that PER (Packet Error Rate) is 1%±0.2%. • To verify that the forward frame rates match the requested DRC. • To Verify that Inter Subnet Active handoffs (ISSHO) are successful • To verify that A13 dormant handoffs are successful

6.2 EVDO Site Shakedown Process The shakedown process is as follows: • Setup the laptop, the AT, GPS, and data collection tool (Optis-M or Invex3G) in the drive test van. • Drive to near (within 1 km) the cell site. • Park the van in a location that is covered only by the alpha sector. • Start the laptop, AT, Optis-M, or Invex3G. Setup the appropriate logmask. • Click on the Auto Call button on Optis-M or right click on the AT device in the workspace of Invex3G

and select “Properties”. • In Optis-M, click on the “Create New Scenario” button. Create an FTP Get scenario (please see the

example below). Then click OK. • In the device properties window of Invex3G, select the “Data Connection” tab and click on the “Add”

button. Select “FTP Get” from the “Task Type” drop down menu. Configure the FTP server IP address in the “Destination Address” box as well as the path and filename, username and password in the appropriate boxes.

Invex3G and Optis-M FTP Get call scenario for shakedowns.

Invex3G

Optis-M


• In Optis-M select the call scenario you just created for the autocall. Select “All scenario” for the

AutoCall Logging Option.

• Start the log by clicking OK.

Invex3G • In Invex3G, all that is required to start logging after the Data Connection configuration is completed is

to go to the “Connections” menu select “Connect All”.

Optis-M - Verification • Check the OPTis-M EVDO AT Status window to make sure that:

o The AT state changes from idle to “connected”. o The session state is “open”.

Invex3G - Verification • In Invex3G verify that:

o The Device color changes from Red to Green

o The “State” or History window shows:

Connection State = Connected Connection Type = 1xEV-DO Session State = Open Neighborlist of the sector is correct RF Channel is correct The Active set PN is correct for the sector under test. Receive diversity is enabled

Optis-M – Scenario Select

Invex3G – Connect All Devices


Optis-M - Verifications • Check the 1xEV-DO Pilot Set Graph window in Optis-M and verify that:

o Alpha sector is the serving sector PN. o The neighbor list of alpha sector is correct. o The channel number is correct.

• Check the Network Status window in Optis-M and verify that: o The AT receive diversity is on.

Invex3G State and History Windows

Optis-M – Network Settings


Examples of OPTis-M display to check during shakedown. • From the EVDO Rate Statistics window, verify that the reverse link is in variable rate.


• Start driving towards beta sector. Monitor the Pilot Pn[ ] and verify the AT does the softer handoff with alpha and beta sector (both PNs are in the active set). Ensure that sector switching to beta sector occurs without a connection drop. Stop the van at a location that is covered only by beta sector. Verify that now only beta sector PN is in the active set.

• From the EVDO Rate Statistics window, verify that PER = Bad CRC/Total CRC = 1%±0.2%. • Stop the log. • Repeat the steps above for:

o Connection origination at beta sector and softer handoff to gamma sector. o Connection origination at gamma sector and softer handoff to alpha sector.

• Fill out the shakedown form (an example is given in Appendix B).

6.3 EVDO Inter Subnet Active Handoff (ISSHO) Shakedown Process • Setup the GPS, AT, Laptop, and Optis-M or Invez3G inside the test van • Drive to the subnet boundary on the source side • Start the laptop, AT and the data collection tool. Select all the EVDO attributes for the log mask.

Optis-M • Click on the Auto Call button in Optis-M and then, click on the “Create New Scenario” button. Create

an FTP Get scenario (please see the example below). Then click OK. • Select the call scenario you just created for the autocall. Select “All scenario” for the AutoCall

Logging Option. • Start the log by clicking OK.

Invex3G • Right Click on the device in Invex3G and select “properties” followed by the data connection tab. • Select “Add” from the “Data Connection” tab and configure an FTP Get task then select OK. • Select the “Connections” menu object then “Connect All”. Logging will start.

Optis-M • Check the OPTis-M EVDO AT Status window to make sure that:

o The AT state changes from idle to connected. o The session state is open.

• Check the 1xEV-DO Pilot Set Graph window and verify that: o The serving PN is on the source side of the subnet boundary

• Begin driving across the subnet boundary toward a site on the target subnet o Verify that a PN from a site on the target subnet is added to the Active set by looking at the

Best ASP SINR, EVDO Finger Information window, and EVDO Airlink Summary window. EVDO Finger Information – Optis-M


EVDO AT Status

Invex3G • Verify from the Invex3G state window that:

o Connected State = Open o Session State = Open

• Open a bar Chart or History window and verify that: o Active set pilot PN is from a DOM on the source RNC

• Begin driving across the subnet or RNC border and verify: o Active set has a PN from the target subnet or RNC is visible in the bar chart or History

window.


• Repeat the process coming from the target subnet side. • Insure that the call does not drop as the subnet boundary is crossed.

6.4 (A13) Dormant Session Handoff Shakedown process • After completing the Inter RNC active handoff (IRAHO) test stop AT logs and proceed to one side of

the subnet / RNC boundary. • RNC logs will need to be collected to verify dormant session handoffs • Process to setup logs:

o RNC logs can be configured using CLI or DO-EMS GUI. The GUI method is given below.

o Open the EMS GUI and from the EMS elements tree, select “Network Databas” DO-RNC

o Click on the RNC(s) involved in the A13 shakedown process o From the RNC Menu, select “Show” “Cards” o Select each RNSM and SC Card followed by “RNSM or SC Card Menu” Show Log

FacilityMgr o Select “LogFacility” Modify

Invex3G


o Insure that : Logging is enabled for the Call Control Component Logging Enabled = True Maximum Severity = 25 Output to File = True Maximum Severity on File = 25 Select Submit Check logging status

o When Logging is complete ftp the log files in bin or GZ format down to a PC for analysis.

NORTEL -07>shell NORTEL-07(shell)(disk0)>cd logs NORTEL-07(shell)(disk0)>ls

• Configure the data collection tool to have the AT perform an FTP GET which will insure an Active session. After one download allow the AT to transition to a dormant state (i.e. No active packet data session).

• Before crossing the border, verify from the “EVDO AT Status” window that the “AT State” goes from Idle Accessing Connected Idle and the “Session State” is Open.

• Start driving towards the subnet border and verify that the Session stays Open but the Subnet ID changes

Optis-M EVDO AT Status


• Verify that UATI, Subnet Mask, and Color Code change when the subnet border is crossed and the

session stays open as the subnet border is crossed. o Note: There are two ways in which the AT can trigger an A13 dormant session handoff:

1. UATI Request using its present UATI - occurs when the AT has continuous reception of the network broadcast messages as it moves from one subnet to the other.

2. UATI Request using RATI – (also referred to as “Prior Session-initiated dormant handoff”), this typically occurs when the AT loses CCH supervision as it crosses from one subnet to the other (e.g. AT powers down in one subnet and powers up in another).

• Repeat the test coming from the opposite side of the subnet boundary.

6.5 Data Processing and Analysis After shakedowns are complete the collected data should be analyzed to insure that call processing functionality and configured data fill parameters are accurate.

6.5.1 EVDO Site Shakedown Analysis After the shakedowns have been completed, process the shakedown files with RF Optimizer. Do the following: • Verify that there is only one connection (i.e., there is no connection drop) by looking at the Statistics

Tool – Calls Statistics.

Invex3G State


Example of shakedown call statistics showing access success and no connection drop.

• Verify that softer handoff occurs. Look at the PN1 and PN2 columns in Grid View and check that the

PNs change from alpha sector PN to alpha and beta sector PNs to beta sector PN only (similarly for beta to gamma and gamma to alpha).

• Verify that average PER = 1%±0.2%. Right click on red bar below the PER column in Grid View and click on Average.

• Verify that the Rx Power is appropriate for the shakedown locations. See the illustration below for an example of softer handoff occurrence, average PER, and Rx Power display.

Example shakedown file analysis showing softer handoff and average PER. • The following RF parameters can be verified from the messages:

Message & Parameters Definition Recomm. Value

Quick Configuration or Sync CHAN_NUM Channel Number for this carrier - Pilot_PN Pilot PN offset for this sector - COLOR_CODE Unique Color Code value for the RNC -

Access Parameters

ACC_CYCLE_DURATION Time instances at which AT may start an access probe 32 ACCESS_SIG ? OPEN_LOOP_ADJ Nominal power used by AT in open loop pwr estimate -86


PROBE_INITIAL_ADJ Correction factor used by AT for the initial tx on Access Ch. 0 PROBE_NUM_STEP Max number of probes for each access probe sequence 8 POWER_STEP Power step size applied to successive probes 6 PREAMBLE_LENGTH Length of preamble in an access probe 2 CAPSULE_LENGTH_MAX Length of capsule in an access probes 4 PERSIST_COUNT A-Persistence vector for controlling contention on the access ch? 0

Sector Parameters

COUNTRY_CODE Mobile Country Code for this sector - SUBNET_MASK Length of the subnet mask 104 LATITUDE Latitude of the sector, in thousandths of a degree - LONGITUDE Longitude of the sector, in thousandths of a degree -

ROUTE_UPDATE_RADIUS Distance beyond which the AT must send Route Update Message 0

(disabled)

REV_LINK_SILENCE_DURATION Duration of the silence interval to be maintained on the reverse link 0

(disabled)

REV_LINK_SILENCE_PERIOD Pperiod of the silence interval 0

(disabled) NEIGHBOR_COUNT Number of neighbors in the NL of this sector - NEIGHBOR PNs PN list of neighbors - NEIGHBOR_SRCH_WIN_SIZE_INC "True" if there are neighbor PNs with different window size set 0 NEIGHBOR_SRCH_WIN_SIZE Search window size of a neighbor. - NEIGHBOR_SRCH_WIN_OFFSET_INC Whether specific search win. offset for neighbors is included in NL 0 NEIGHBOR_SRCH_WIN_OFFSET Search window offset of a neighbor. -

Traffic Channel Assignment

DRC_LENGTH Number of consecutive slots in which the DRC value is repeated 4 DRC_CHANNEL_GAIN Power of DRC Channel relative to Pilot 58 (-6) ACK_CHANNEL_GAIN Power of Ack Channel relative to RL Pilot 6 RAB_LENGTH Length of Reverse Activity Bits for this sector 32

RAB_OFFSET For sectors in same cell should be >= (RABLength 8) slots if possible -

Pilot Sets v2

ASET_WINDOW Search window size for active set 60 chips RSET_WINDOW Search window size for remaining set 100 chips (Neighbor) WindowSize Search window size for neighbor set 100 chips

State Information AT State Tells if the AT is connected to the access network - Session State Tells whether the EVDO session is open or closed - Idle State Tells if the AT is active or inactive in an idle state - Connected State Tells if an AT connection to the access network is open or closed - Route Update State Tells if the mobile is connected to the traffic channel -

HDR_Hybrid_Mode Hybrid or EVDO (HDR) is active On =

(hybrid)


Sector Parameters 42201501 BAND : 1 CHAN_NUM : 1125 Pilot_PN : 112 HSTR : 0 SEQ_VALID : 0 ACK_SEQ_VALID : 0 FRGMENTED : 0 RELIABLE : 0 SEQ_NO : 255 ACK_SEQ_NO : 255 InConfPro : 0 Type : 15 COUNTRY_CODE : 1 SECTOR_ID_1 : 0 SECTOR_ID_2 : 16777292 SUBNET_MASK : 104 SECTOR_SIG : 1 LATITUDE : 653025 LONGITUDE : 7297592 ROUTE_UPDATE_RADIUS : 0 LEAP_SECONDS : 13 LOCAL_TIME_OFFSET : 1808 REV_LINK_SILENCE_DURATION : 0 REV_LINK_SILENCE_PERIOD : 3 CHANNEL_COUNT : 1 [ 1 ] SYSTEM_TYPE2 : 0 BAND_CLASS2 : 1 CHANNEL_NUMBER2 : 1125 NEIGHBOR_COUNT : 9 [ 1 ] PILOT_PN : 104 [ 2 ] PILOT_PN : 108 [ 3 ] PILOT_PN : 116 [ 4 ] PILOT_PN : 120 [ 5 ] PILOT_PN : 124 [ 6 ] PILOT_PN : 132 [ 7 ] PILOT_PN : 148 [ 8 ] PILOT_PN : 152 [ 9 ] PILOT_PN : 156 [ 1 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 2 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 3 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 4 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 5 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 6 ] CHANNEL_INC : 1


SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 7 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 8 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 [ 9 ] CHANNEL_INC : 1 SYSTEM_TYPE : 0 BAND_CLASS : 1 CHANNEL_NUMBER : 1125 NEIGHBOR_SRCH_WIN_SIZE_INC : 0 [ 1 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 2 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 3 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 4 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 5 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 6 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 7 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 8 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found [ 9 ] NEIGHBOR_SRCH_WIN_SIZE : Not Found NEIGHBOR_SRCH_WIN_OFFSET_INC : 0 [ 1 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 2 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 3 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 4 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 5 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 6 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 7 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 8 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found [ 9 ] NEIGHBOR_SRCH_WIN_OFFSET : Not Found

6.5.2 Inter RNC Active Handoff Shakedown Analysis After completing the Inter subnet handoff runs an analysis of the collected logs should be performed to determine rate of success. • Post process collected logs with EVDO RFO • Verify from the statistics view there was only one connection for each run. • If there were any dropped connections, a determination of the cause should be made. • Open the Grid view and verify that the AT was in soft handoff with PN(1) from sector with the source

RNC as a primary and PN(2) from sector with the source RNC as a secondary.


Figure – Example of Grid View showing ISSHO

6.5.3 A13 Dormant Session Handoff Analysis

• Process RNC logs from A13 session handoff shakedowns with NEWTUN. • Verify from the logs that a UATI Request initiated handoff triggers a A13 Request and finally that an

A13 Confirm is sent indicating that the session is successfully handed off from the source suibnet to the target subnet. (See Figure Below).

Example: RNC Logs showing A13 handoff (Foreign UATI)

PN 116 is Primary and PN 164 was homed as secondary


7 Golden Value (Stationary) Testing

7.1 Introduction The objective of the Golden Value Testing (i.e. stationary throughput testing) is to measure the maximum user application throughput in the forward link using the network components (PDSN, DO-RNC, DOM, FTP server, AT, and laptop) that will be used in the cluster optimization stage. During this exercise, the TCP and other settings can be optimized to improve the application performance. Once those settings are optimized and finalized, the measured user application throughput becomes the “golden” (or baseline) value, i.e., the upper bound of what we can achieve in the network. This measurement should be done in a stationary location with a good RF link (close to the cell site, SINR ≥ 6 dB or Ec/Io ≥ -1 dB) and covered by a single sector. The user application throughput is measured with FTP test using a single AT. Since we want to measure the maximum user application throughput, we need to ensure that the network is unloaded (no other users in the network). For the stationary golden value testing it is recommended to perform 10 downloads of a 5MB test file. The test file should be a zip file or MP3 to insure that any compression algorithms do not skew the results..

7.2 FTP Server Configuration To maintain a controlled test environment, it is recommended to connect the FTP server to the same subnet as the PDSN. It is desirable to avoid having the test data traffic going through routers, firewalls, etc., to avoid unpredicted problems like delay that will degrade the performance.

7.3 Golden Value Testing Process The Golden Value Testing process is as follows: • Setup the laptop, the AT and the data collection tool (Optis-M or Invex3G) in a stationary location

with a good RF link. • Start the laptop, AT and the data collection tool. Setup the appropriate logmask.

Optis-M – FTP Setup • Click on the Auto Call button, then click on the “Create New Scenario” button. Create a FTP Get

scenario similar to the one used for shakedown (see section 4.2). • Select the call scenario you just created for the autocall. Select “All scenario” for the AutoCall

Logging Option. • Start the log.

Invex3G – FTP Setup • If an FTP GET scenario was already configured for shakedown process just select “Connections” and

then “Connect All” to start the logs Connection and FTP State Verification

• Check the OPTis-M EVDO AT Status or the Invex3G state window to make sure that: o The AT state changes from idle to connected. o Verify that in Optis-M that the “Current State” in the CDMA Statistics window = FTP DN

Start o Verify in Invex3G state window that the “FTP Get” task state = “Running” o The session state is open.

• Check the Optis-M “Network Status” window or Invex3G “State” window and verify that: o The AT receive diversity is on / Enabled.

Optis-M – Throughput and C/I Verification • Check the EVDO Signal Graph display in Optis-M and verify that:

o Best ASP SINR is good (mostly above 6 dB). o DRC Requested are mostly 1.2288 Mbps or higher.


o Slot utilization is high (in 80% - 90% range) due to single user and full-queue condition. • Check the reverse traffic statistics in EVDO Rate Statistics window and verify that the average reverse

data rate is sufficient to support the forward link data rate [2]. • Verify that the Application Throughput Rx is consistent with the data rate distribution shown in the

Forward Traffic Statistics in EVDO Rate Statistics window, and also with the average DRC and Slot Utilization. Since the SINR in this location is good, RLP retransmission should be minimal, and an Application Throughput Rx should not be much less than 80% of the average physical layer throughput (the overhead being about 20%; please see [11]). Lower throughput may indicate TCP problem. See Section 4.4 (TCP/IP Performance)

Optis-M EVDO Signal Graph display. Invex3G – Throughput and C/I Verification • Check the Invex3G State window and verify the following:

o Composite C/I is > 6dB o DRC requests “DRC Value” are >= 1.2288Mbps o Task RX Throughput (kbps) (Cumulative) is consistent with the DRC request Rate o Packet Error rate <1% o Reverse link throughput is sufficient to sustain forward link (34.9kbps RL for 2.4Mbps FL). o Slot Utilization is between 80 and 90% o RPC Channels in use (Users connected to sector) = 1


• Stop the logs. • This test should be repeated five to ten times (or as much as time and resources permit) to ensure that

the result is statistically valid.

Invex3G State Window


7.4 Data Processing and Analysis After the Golden Value Testing is done, process the files with RF Optimizer. Do the following • Open the Graph View. Display the DRC Requested and Traffic C2I. Verify that DRC Requested

follows the trend of C2I1. • Also display App Rx Rate, Actual Fwd Data Rate, and Fwd Serving Rate in another window. Verify

that Fwd Serving Rate follows DRC Requested. This proves that the network follows the DRC requests from the AT.

Example of the golden value testing result. (blue = 1st parameter, green = 2nd parameter, red = 3rd parameter)

• Click on the Statistics Tool → Data Statistics → Application. Calculate:

Average user application throughput = Rx Data (Bytes) x 8 / Elapsed Time In the example below: Average user application throughput = 15,000,000 x 8 / 62.683 = 1.91 Mbps


• Verify that the average user application throughput obtained in this test is appropriate for the SINR

(C2I) condition. The following plot can be used as a guide. Please refer to [1] to more information on the throughput performance number.

0

400

800

1200

1600

2000

2400

-8 -6 -4 -2 0 2 4 6 8 10 12 14

Median Ec/Nt (dB)

Thro

ughp

ut (k

b/s

0.826 km/h - 1 Ant. 0.826 km/h - 2 Ant.

3 km/h - 1 Ant. 3 km/h - 2 Ant.

120 km/h - 1 Ant. 120 km/h - 2 Ant.

Single user throughput vs. Ec/Nt.


8 OM Collection and Analysis

This chapter discusses methods for collecting and analyzing operational metrics for purposes of performance benchmarking. 1xEVDO system operation measurements have the following granularity:

• Per RNC • Per Slot (slot corresponds to the physical RNSM slot in the RNC chassis) • Per DOM (RN) • Per Sector Carrier • Per Connection • Per Interface (physical or logical) • Per Channel Operational measurements should be collected during the baseline drive test as well as during subsequent optimization cluster drive rounds of the network to get a system level view of the performance attributes given below.

8.1 Connection Attempts There are a few varieties of connection setup attempts. These attempts can occur during various stages of connection and are denoted as “Total Served Connections”. The following events will result in a successful connection attempt. These OMs are per RNC level metrics.

DO-RNC has a connection open – occurs when a connection request is received from the AT while a current connection is already open. In this case the Connection request is cached until the current connection can be closed. The following OMs are pegged in the case: numConnectionReqsWhileOpen and numConnectionRequestsfromAT

DO RNC is in the process of tearing down a connection – in this case the connection request is cached and the previous connection is allowed to close normally before the new request is processed. The following OMs are pegged in this case: numConnectionReqsWhiletearingdown and numConnectionRequestsfrom AT

DO RNC is in the process of setting up a connection – a connection request is received by the RNC while the network is waiting for a TCC from the AT. In this case the previous TCA is retransmitted to the AT. The following OMs are pegged in this event: numConnectionReqsWhileSettingUp and numConnectionRequestsfrom AT

Total Connection Setup Performance OMs can be found in section 7-4 of [17]

8.2 Failed Call Attempts (FCA / Access Failure) Failed connection attempts are defined as events where the AT does not successfully arrive on the traffic channel. Metrics are defined to measure the rate of failed call attempts both before and after a demarcation point. Performance measured before the demarcation point is defined as; connections attempted after session configuration has taken place but the AT has no data to send, therefore A10 is not opened (no data to send). Performance after the demarcation point is defined as: connection setups attempted after session configuration is complete and the A10 connection is opened.

Failed Call Attempts OMs are given in section 7-5 of [17].

8.3 Dropped Connections (Abnormal Connection Closes) A connection drop refers to an abnormal connection close. These abnormal connection closes can occur as a result of RF link loss, Reverse Link Handoff Failure, or a variety of other reasons. Connection Drop or Abnormal Connection Close metrics are given in section 8-4 of [17].


8.4 Session Setup Performance (includes A13 session handoffs) Session setup is the first stage of communication between the AT and the AN, and is usually initiated by a UATIRequest message from the AT. The EVDO session setup usually involves the following stages:

• UATI Request • UATI Assignment • A13 dormant session Handoff • Session configuration • Terminal authentication • Hardware ID retrieval • AT ID validation

Total 1xEV-DO Session Setup Success Rate OMs are defined in section 2-8 of [17]

8.4.1 UATI Request Initiated A13 Dormant Handoff Performance Regular A13 handoff is triggered when an AT crosses the subnet border without losing supervision of the DO carrier.

UATIRequest Initiated A13 Dormant Handoff OMs are defined in section 2-11 in [17]

8.4.2 Prior Session Initiated Dormant Handoff Performance A prior session initiated dormant handoff occurs when the AT, during the session configuration phase requests to retrieve a prior session from the RNC or when the AT looses supervision of the source RNC and regains it on the target RNC. In this situation the target RNC will attempt to retrieve the prior session information from the source RNC over the A13 interface. For session transfer to proceed, A13 dormant handoff must be enabled on the target RNC.

Prior Session Initiated Dormant Handoff OM’s can be found in section 2-13 of [17].

8.4.3 A10 Connection Setup Performance In the previous releases, an A10 connection was setup every time the AT completed a location notification or after every session configuration regardless of the need to send data or not. In release 3.0 we now have a UA10 minimization feature that prevents the setting up needless A10 connections. The minimization feature looks for the following events to trigger the opening of an A10 connection. Otherwise A10 remains closed.

ULN ModeOpen A10 if

New Session ULN receivedInter RNC Dormant Handoff (Regular) Non-0 PDSN IP in A13 Response OR ULN ReceivedInter RNC Dormant Handoff (Prior Session) ULN ReceivedInter Tehnology handoff intra RNC ULN ReceivedInter Tehnology handoff inter RNC ULN Received

MIP ModeOpen A10 if

New Session First RLP Packet received from AT (a.k.a PPP resync)Inter RNC Dormant Handoff (Regular) Non-0 PDSN IP in A13 Response OR PPP ResyncInter RNC Dormant Handoff (Prior Session) PPP resyncInter Tehnology handoff intra RNC PPP resyncInter Tehnology handoff inter RNC PPP resync

Refer to section 6-1 of [18] for A10 Connection Setup Performance OMs


8.4.4 Session Configuration Failures There are two phase of session configuration; the AT initiated portion where the AT initiates negotiation of protocol attributes, and the AN initiated portion where the AN initiates negotiation of protocol attributes. The session configuration process is not complete until both the AT and the AN have committed to the recommended protocol attributes. Session configuration is unsuccessful if: the protocol negotiation fails between the AT and the AN, the session configuration timer expires (60secs), or the DO Airlink connection drops during the process.

Session Configuration Failure metrics can be found in section 5.1 of [17].

8.5 Serving Sector Switching and Soft/Softer Handoff Performance In 1xEV-DO there is no soft handoff on the forward link. The AT indicates its desire to switch sectors to the AN by sending a null DRC value and thus the AN handles the sector switching. On the reverse link there is soft and softer handoff just like in 1xRTT.

Refer to section 11-1 of [17] for Serving Sector Switching and Soft/Softer Handoff Performance OMs.

8.6 Inter RNC Active Handoff Traffic Statistics and Secondary Border Measurement Prior to release 3.0 there was no mechanism for preventing call drops at subnet (RNC) boundaries. In prior releases, when the AT would send a route update containing a Pilot from another subnet, the Pilot Lookup would fail (RNC has no knowledge of pilots in other subnets) and the connection could drop as a consequence.

With the introduction of release 3.0, the Inter RNC Active Handoff feature will be added. This feature will allow DOMs from adjacent RNC to be secondarily homed to the anchor or source RNC in the immediate subnet. This capability will allow active calls to continue on DOMs in adjacent subnets. New OMs are available to help answer the following questions:

• Should the DOM be homed to other secondary RNCs to extend the border region? • Should the DOM not home with any secondary RNCs it is currently configured with? • Is the DOM homed to the optimum “Primary” RNC? Refer to section 13-8 of [17] for description of Inter RNC Active Handoff Traffic Statistics and Secondary Border Measurement OMs.

8.7 1xEVDO Paging Performance Measurements Prior to release 3.0 the system would only send a single page to the AT. Current with release 3.0 is the DO repage feature. This feature can be configured to repage an AT up to 2 times (total of 3 pages to AT) before declaring an airlink paging failure. Additional OMs have been added in release 3.0 to account for the repage feature.

When the DO RNC attempts to set up a connection with the AT it can do a normal setup involving paging the AT or by Fast Connect which does not involve paging the AT. The OMs presented for this section involve pages that are actually sent to the AT from the AN.

Refer to Section 10.1 of [17] for definitions of Paging Performance OMs.

8.8 Per Sector Airlink Resources Allocation (Blocking) Airlink resource allocations refer to the resources required for the AT to add pilots to its Active Set at connection setup or during Soft / Softer Handoff. These OMs help during network deployment to determine where resources may limit connection setup success. For example, if during connection setup or Soft / Softer handoff the Route update message requests the addition of pilot P1, P2, and P3. The RNC will send allocation requests for the requested pilots. If DOMs are able to successfully allocate resources for P1 and


P3 but a traffic channel can not be assigned for P2, a block due to Reason X will be pegged against Pilot P2 only and success counts will be pegged against pilots P1 and P3. In the end the connection setup will fail. These OMs measure Airlink resources allocation for primary and secondary DOMs during connection setup and soft / softer handoff.

Refer to Section 13.1 of [17] for more details on Per Sector Airlink Resources Allocation OMs.

9 Cluster Drive Testing

This chapter discusses the typical cluster optimization drive testing that is done prior to acceptance testing. The objective of cluster drive testing is to measure the 1xEV-DO network performance in mobility conditions. Typical KPIs measured are connection drops, access failures, connection setup time, and mobility throughput. Prior to the optimization drive rounds a baseline drive test should be performed to determine the initial performance capability of the system. The baseline will provide an initial look at where problematic areas are such that optimization priorities can be established. For this test, the network should be divided into clusters; each containing 10-15 cell sites. The same setting (laptop, AT, FTP server) used in the stationary data testing should be used for cluster optimization. The cluster optimization is done using one laptop, data collection tool (Optis-M or Invex3G) and two ATs in a drive test van. AT #1 will perform FTP download of a small file (100 KB) to measure the access failures and connection setup time. AT #2 will perform FTP download of a large file to measure the connection drops, forward-link sector switching time, and throughput. There are two options for the file size for AT #2, 5 MB or 100 MB. 5 MB file will yield about 80 s calls at 500 kbps average throughput. In this case, to calculate the connection drop rate, we just take the number of connection drops divided by the number of calls. In the second option (100 MB), divide the total call time by the average CHT (call holding time) to get the number of calls. Then, to calculate the connection drop rate, take the number of connection drops divided by the number of calls.

9.1 Data Collection Process The data collection process for cluster optimization is as follows: • Setup the laptop, the two ATs and Optis-M or Invex3G at the drive test route starting point. • Start the laptop, AT and data collection tool. Setup the logmask to collect all EVDO attributes

Optis-M – Scenario Setup • Click on the Auto Call button then click on the “Create New Scenario” button. Create an FTP Get

scenario for Phone 1 (AT #1) and Phone 3 (AT #2) according to the example below. Then click OK. • Verify the following settings:

o “Allow Dormant State” = Checked o “Keep PPP” = Checked. This will insure that the dialer is not disconnected between data calls. o Dormant Mode timer (Not Shown Below) = at least 15sec. This will insure that at the end of

each data call the AT will transition to a dormant state before the next data connection attempt.

o Phone Reset = checked. This if the phone fails to connect after the defined number (None Success) the Optis Client will reset the phone automatically before attempting to connect again.

• None Success = 2 o Retry FTP in Server Drop = Checked If an FTP server failure is detected, Optis-M will

attempt to re establish the FTP connection


Optis-M Auto Call setup for cluster drive testing. • Select the call scenario you just created for the autocall. Select “All scenario” for the AutoCall

Logging Option. This will allow data from both scenarios (ATs) to be captured in one log file. • Start the log by clicking OK.

Invex3G – Scenario Setup • For EVDO data collection it is imperative that the Invex3G Chassis be configured with at least 2 DI

boards. The legacy CI boards are processor limited and will result in a reduction in throughput performance.

• Right Click on the device (AT#1) and select Properties. Select the Data Connection tab to configure the data connection properties.

• Follow the same process to configure (AT#2). See figures below for Reference. o Disconnect Timer = 15 seconds For Disconnect tasks, enter a Hang up Timeout value as

required. If Invex3G does not see data from the network for a certain length of time, it assumes that the data task is dead and tries to hang up the call (session). Invex3G waits for an acknowledgement from the network (LCP TERM ACK); however, at some point the software gives up waiting for the acknowledgement and starts another task loop. If the acknowledgement is not received, it is possible that the network will keep the original data session open. When a new task loop is started by Invex3G, the network may not allow the new session since it is keeping the previous session open. The Hang up Timeout determines the time that Invex3G will keep trying to get a hang up acknowledgment from the network to confirm that the previous data session was cleared. If you are experiencing call backs from the network when a session is ended, or the network will not allow new sessions, try increasing the Hang up Timeout value to a larger number.

o Wait Timer = 5 seconds time between calls where phone will be inactive. o Initial Connect - Parameters required for connection to the EVDO network. See the “Data

Connection Task example below. Connection Type = EVDO Authentication = Chap / PAP Compression = None TCP Input (RX) window = default


Serial Layer MTU = default If network is simple IP, set the Username and Password If network is Mobile IP Username and Password should be left blank.

• After the initial EVDO connection parameters are configured select the “Add” button at the bottom of the “Data Connection” window to add the FTP GET task

o Task Name – For AT#1 should be Access Failure Performance for AT#2 Dropped Call Performance

o Task Type – FTP GET o Destination Address – IP address of FTP server o Path and Filename – The path on the FTP Server, starting with the root directory, where the

test file is loaded. A 5MByte file should be used fort AT#2 doing dropped connection and throughput

testing A 100kbyte file should be used for AT#1 doing access failure testing.

o Username and Password – As required to access the Applications Server

Invex Initial Connection

Config.


• After all data connection tasks have been defined click ok. Go to file and save workspace as “Cluster

Drive.IWF” • Select “Connect All” from the connections menu then “Record logfile” from the file menu to start logs.

Drive Testing Process • Start driving along the cluster drive test route. • Check the following display in Optis-M throughout the drive test:

o GPS Status window: to ensure that GPS is working. o 1xEV-DO Pilot Set Graph: to monitor that the AT does soft handoff as necessary with other

sites. o EVDO Signal Graph and Call Statistics window: to monitor the data call progress and the call

statistics. • Check the following displays in Invex3G throughout the drive test:

o Go to displays and select “New” and add a State display and a 1D bar Chart. o In the state display drag the “Current Task Status” from the data connection Codec of each

device listed in the device tree. o The Current Task Status will list the following attributes:

Connection State - Verify Connected Connection Type – Verify 1xEV-DO Task State – Verify Running Task Name – Verify FTP GET 5MB or 100kbyte Task RX Throughput (kbits/s) (Cumulative) – Verify > 0 kbits/s

o In the Bar Chart drag the “Satellites” icon from the GPS Codec. Verify GPS lock to at least 7 satellites

• Verify from the Reports icon in the device tree that Alerts configured to show: o Dropped Calls o Call Set up Failures o Data Connection Failures o Data Task Faliures

• If any of these events occur there will be an audible alarm as well as a recoding of the event in the Alerts tab of the Invex3G workspace. (See example Below)

Invex Data Connection

Configuration


Call Statistics window in Optis-M.

Invex3G State Window • Note down the areas where access failures, connection drops, and/or handoff failures occur. • At the end of the drive test, stop the logs. Try to keep each log to be no more than 30-minute long.


9.2 Data Processing and Analysis After the data collection for the cluster optimization is done, process the AT log files with RFO and the RNC logs files with NEWTUN.

9.2.1 Access Failure Rate An access failure is defined as the number of times (during the cluster drive test) the AT failed to arrive at the traffic channel. It should measure the RF-related failures excluding the ones caused by resource blocking and other non-RF related failures.

= AttemptsConnectionValid

TConArrivetoFailsATAccessFail __#_____#%

A successful access attempt is marked by Connection Request message. AT success on arriving on the traffic channel is marked by Traffic Channel Complete message. Access Failure Rate from RNC Logs • After processing the RNC log files in NEWTUN select the files from the Data Manager window.

The call statistics from the RNC logs will be displayed in Statistics view. • Call statistics are grouped as follows in NEWTUN:

o Network Statistics – statistics for calls displayed by RNC, IP, and PN. o Connection Details – Call Summary for every connection identified in the logs o Access Terminal – call statistics that are displayed according to ESN, IMSI, or Hardware

type. • For each group of statistics there are groupings of statistics as follows:

o Data = statistics for Active Data Calls o Session = statistics for session configuration related connections o All = (Data + Session)

• The statistics shown in these views will be for all users on the network. There are utilities in NEWTUN that will allow engineers to calculate failure statistics for the AT under test.

• Select the Access Terminal tab then click on the data icon.

Connection Request (Access)

AC Ack (Control)


(Control)

RTC Ack(Fwd Traffic)

Traffic Channel Complete

(Rev Traffic)AF


AC Ack (Control)


(Control)



(Rev Traffic)AF


AC Ack (Control)


(Control)



(Rev Traffic)


AC Ack (Control)


(Control)



(Rev Traffic)


AC Ack (Control)


(Control)



(Rev Traffic)


AC Ack (Control)


(Control)



(Rev Traffic)AF


• Sort the list by IMSI

• To determine the access failure rate, look for the IMSI of AT#1 (AT used for Access Failure

Testing.

• The Access Failure Rate for the AT inclusive of only Data or Session Configuration related

connections is defined as:

( ) 100% XAccessesIncompleteBlocksAttemptsfailuresAF

+−=

• Note: Incomplete Access connections that reached the access phase, but had no more messaging to

indicate the outcome. • Click on the Sessions icon and sort by IMSI to determine the Session related Access Failures.

• The get the total access failure rate click on the “All:” icon.

• The Total Access Failure Rate is defined as:

( ) 100% XspleteAccesTotalIncomsTotalBlockttemptsTotalCallAdCallsTotalFaileAF

+−=

• For more information on which calls failed, click on the Total Failed Calls in the statistics view.

The actual calls will be listed in the Active List.


• By clicking on the calls in the Active list the engineer will open the message view where the actual

RNC level messages will be highlighted in green in the message view.

• Connection was closed because there was an unknown UATI (Local Connection Close, Reason

Code 2). Local Close Event Reasons (RNC Logs):

Active Call List

When all RTC comms. are lost

DLSM All RTC Lost9

Forward Traffic Channel lostDLSM No FTC10

Unable to request HDR Fast Path to Stop or re Start transmission of data

DLSM HFP Error11

Generated by CSM when a connection request is received when the AN thinks it already has a connection open. ( No longer used)

Connection Request Error Close

8

Called by SCSM when Session configuration is complete

Session Configuration Close

7

Not currently usedBTS Requested Close6

Sent by Call Control when the connection has not passed any data on the FWD or RVS link for a configurable inactivity period.

Call Control Inactivity Close

5

When HDR Fast path encounters errors.

HDR Fast Path Close4

When HDR slow path encounters errors

HDR Slow Path Close3

SSM to close connection and return to disabled state

Session Close Disable2

Called by SSM to close connection.

Session Close1

CommentsReason DescriptionReason Code

When all RTC comms. are lost

DLSM All RTC Lost9

Forward Traffic Channel lostDLSM No FTC10

Unable to request HDR Fast Path to Stop or re Start transmission of data

DLSM HFP Error11

Generated by CSM when a connection request is received when the AN thinks it already has a connection open. ( No longer used)

Connection Request Error Close

8

Called by SCSM when Session configuration is complete

Session Configuration Close

7

Not currently usedBTS Requested Close6

Sent by Call Control when the connection has not passed any data on the FWD or RVS link for a configurable inactivity period.

Call Control Inactivity Close

5

When HDR Fast path encounters errors.

HDR Fast Path Close4

When HDR slow path encounters errors

HDR Slow Path Close3

SSM to close connection and return to disabled state

Session Close Disable2

Called by SSM to close connection.

Session Close1

CommentsReason DescriptionReason Code


Access Failure Rate from AT logs • Post Process the AT logs from the Cluster Drive using EVDO RFO • Open the Statistics View and select Call Statistics.

Example of Access Failure statistics in EVDO RFO • Example using the numbers above: • Access Failure Rate

o = DO Access Failures / (DO Attempts – DO Blocks – Handoff to 1x – DO EOF Partial Access)

o = DO Access Failures / (DO Access Failures + DO Access Successes) o = 3 / 25 = 12%

The access failure rate should meet the optimization criteria. Please see troubleshooting section for guidance on the typical access failure rate value and methods for troubleshooting if it is not achieved.

RFO Message View window showing one complete call.

9.2.2 Connection Drop Rate A normal call is ended by a Connection Close message from the AT with the reason = Normal Close or a Connection Close from the AN with the reason = Normal Close. Any other call termination is considered a


connection drop (e.g. forward Connection Close message with reason = Connection Error or QCP Connection Release packet with reason = System Lost). Dropped Connection Rate from RNC logs • After processing the logs with NEWTUN, open the Data Manager window and select the database

the files where processed to. • After opening the data base the statistics view will open. As stated above, call statistics are divided

into three categories: o Network Statistics – statistics for calls displayed by RNC, IP, and PN. o Connection Details – Call Summary for every connection identified in the logs o Access Terminal – call statistics that are displayed according to ESN, IMSI, or Hardware

type. • Each of the above categories are further grouped by call type for more granularity:

o All – includes all calls o Data – connections setup for an active data session o Session – connections setup for EVDO session configuration

• Select the Access Terminal tab from the call statistics view and sort the statistics by IMSI.

NEWTUN Access Terminal Statistics View Sorted by IMSI • The data presented in the file will be representative of all ATs in the network at the time of testing. • Find the IMSI of AT#2. The statistics on this line will be used to calculate the dropped connection

rate. • Dropped connection statistics should be calculated for individually for Data, Session, and All

connection types according to the following equation.

100% XTrafficIncompleteAccessesSuccessfulDropsnectionDroppedCon

−=

• Note: Incomplete Traffic – is defined as a connection that made it to the traffic channel but there

were no additional messages to indicate the outcome (i.e. if there was a normal or abnormal connection close).

• For more information on why connections were dropped double click on Total Dropped Calls in the statistics view. This will bring up all of the calls in the Active list.

• As above, by clicking on each call in the Active List, RNC logging messages for that call will be displayed in the message view.


NEWTUN Message View • Call dropped due to loss of the Reverse Traffic Channel (Local Connection Close, Reason Code

9). • Another useful method for diagnosing reasons for a call failure is by clicking on the Connection

Details Connection List icon in the statistics window. • From the Connection list, drag the IMSI column header to the black space and then sort the list by

failure reason (i.e. Unclassified Drop). This will list all connections that failed for the selected reason and will be listed in order of UATI.


NEWTUN Connection List • Once the particular failure is isolated, find the IMSI of the AT used for testing. • Expand the view for that IMSI. All the failures meeting the criteria used to sort the list are listed

under the selected IMSI. • Various Attributes are available in this view that may give clues as to why the failure occurred.

The available attributes are selectable by using the attribute selector. • By double clicking on each connection under the IMSI the RNC message details will be shown

message view. Dropped Connections From AT logs • In RFO, open the Statistics Tool Calls to find the number of connection drops. Example:

Example of Connection Drop statistics in RFO • To calculate the connection drop rate:

Method 1: File Size = 5 MB Connection Drop Rate = DO Drops / (DO Access Success – DO EOF Release – Handoff to 1x) = DO Drops / (DO Drops + DO Completions) Example using the numbers shown above: Connection Drop Rate = 7 / (7 + 15) = 31.8%

Method 2: File Size ≥100 MB


Connection Drop Rate = DO Drops/Number of Effective Calls = DO Drops/(Total Rx Time/Call Holding Time) The CHT (Call Holding Time) should be mutually agreed by Nortel Networks and the customer. The Total Rx Time can be found in RFO from Call Statistics Data Statistics RLP.

• The connection drop rate should meet the optimization criteria. Please see troubleshooting section

for guidance on the typical connection drop rate value and ways to troubleshoot if it is not achieved.

• The illustration below shows a connection drop that happened due to poor C2I.

Example of a connection drop in RFO.

9.2.3 Throughput in Mobility Condition Only files/data from AT#2 are used to calculate throughput. There are no throughput statistics available from the RNC logs therefore, all per user mobility statistics will come from AT logs. • After post processing the AT log files with EVDO RFO, click on the Statistics Tool → Data

Statistics → Application. • Look for Rx Active Throughput in the statistics list. This is the average user application

throughput. • EVDO RFO displays the application throughput for each FTP download as well as the average

throughput for each drive test file.


Application throughput numbers in RFO. As in the stationary data testing analysis, check the following to verify that the throughput result is valid: • Use the Graph View to display Traffic C2I, PER, DRC Requested, and Fwd Serving Rate. • Verify that DRC Requested follows C2I 1 trend. • Verify that PER is 1%±0.2% most of the time, except when C2I 1 is bad. • Verify that Fwd Physical Data RX Rate follows the DRC Requested trend. • Verify that the user application throughput obtained in this test is appropriate for the SINR (C2I)

condition. Use the plot in section 7.4 as a guide. Please refer to [1] to more information on the throughput performance number.

• Display the Reverse Data Info in Grid View or Graph View. Verify that the reverse link data rate is sufficient to support the forward link data rate.

Example of DRC Request, App. RX Rate, and Forward Physical RX Rate


Example – RFO Grid View of Traffic C21, DRC Request Rate, Forward Physical RX Rate, etc • The user application throughput under mobility condition here has to meet the optimization

criteria. The throughput number for cluster drive testing will generally be lower than the one obtained in the stationary data testing due to the mobility condition (as shown in the picture in section 7.4). The areas with access failures, connection drops, handoff problems, and coverage problems (low SINR) will also contribute to the decrease in throughput number. Therefore, fixing all of those problems and improving the SINR condition will improve the throughput number. Please see section 10 for guidance on the typical per-user mobility throughput value and ways to troubleshoot if it is not achieved.


9.2.4 Connection Setup Time Only files/data from AT#1 are used to calculate the connection setup time. Connection setup time is calculated from the time AT sends a Connection Request message to the time it sends a Traffic Channel Complete message. In RFO, click on the Statistics Tool → Connection Statistics Connections. Export the table to Excel and calculate the average Time Delta to get the average Connection Setup Time throughout the drive test.

Example of Connection Statistics in RFO The connection setup time should meet the optimization criteria. Please see section 10 for guidance on the typical connection setup time value and ways to troubleshoot if it is not achieved.


10 Troubleshooting

10.1 Session Setup and A13 Handoff Success Rate EVDO Session setup is simply defined as the initial interaction between the AT and the AN. An EVDO session setup is analogous to a registration in CDMA. Session setup is always an AT initiated action. Only a UATI Request message from the AT with either RATI or foreign UATI can initiate a valid EVDO session setup attempt. In release 3.0 A13 dormant session transfer protocol is introduced and will impact the session setup success. EVDO session setup can be classified as follows:

• Regular 1xEV-DO session setup initiated with an RATI (also known as regular session setup) • UATIRequest initiated dormant handoff (also known as Regular A13-dormant handoff) • Prior session-initiated dormant handoff EVDO Session setup consists of the following Stages: • Session instance creation • UATI assignment • A13 dormant session handoff (for dormant handoff only) • Session configuration • Terminal authentication, if enabled • Hardware ID retrieval • AT ID (IMSI and Hardware ID) validation

10.1.1 Data Collection Methods 10.1.1.1 AT Logs

Method of Collection 1. Set AT to download 1MB file – wait 15s. 2. Close AT session from RNC every 60 s. 3. To close the session, at the CLI prompt type > no open 1xevdo session <UATI> 4. Set up Optis-M or Invex3G Log masks to collect all EVDO log attributes. 5. To test A13, setup a short data call and allow it to transition to a dormant state. 6. Drive across the RNC border.

10.1.1.2 RNC Logs

Method of Collection 1. RNC logs should be collected in conjunction with the AT logs to capture Session setup events and

the reason codes associated with any failures. 2. The following shows session setup events with and without A13 dormant handoff.


Regular Session Setup without A13 Handoff

UATI Assignment

UATI Complete

Close Traffic Channel

AT AN

UATI Request

Open Traffic Channel

Session Configuration

Page

Open Traffic Channel

Term Auth, HID, LUP

PDSN

Open A10 (If data to send)

PPP Traffic


ATSource

ANTarget

ANUATI Request (foreign)

UATI Complete

Open A10

PPP Traffic

A13-Request

A13-Response

(with PDSN IP and Hdw Id)UATI Assignment

Location Assignment

Location complete

Skipping Hdw Id exchange

Close A10A13-Confirm

Source closes DO session

PDSNATSource

ANTarget

ANUATI Request (foreign)

UATI Complete

Open A10

PPP Traffic

A13-Request

A13-Response

(with PDSN IP and Hdw Id)UATI Assignment

Location Assignment

Location complete

Skipping Hdw Id exchange

Close A10A13-Confirm

Source closes DO session

PDSN

A13 Dormant Handoff Session Setup

3. Session setup success rate is defined as:

Session Setup Successes = Normal Session Setup Successes + Dormant Handoff Successes + Dormant Handoff Successes Prior Session

Session Setup Success / Session Setup Attempts = Transitioned from SSMWaitForAtIdRsp state to SSMWaitForLocationNotify state / Transitioned from SSMWaitForUATIReq state to SSMWaitForUATIRspFromRC state

4. Set up the RNC logs to capture Call Control information at severity level 16. The events shown in

yellow are what is required to determine the Session Success Rate.

5. The other events shown are useful for determining why a Session Setup may have failed.

Severity Level

Component Debug DO-RNC CPUs Message ID

Description

16 Call Control 011 RNSM and SC 81 Session State Transitions (UATI Req - HwId Resp)

16 Call Control 011 RNSM and SC 151 Session Configuration Event Succeeded

19 Call Control 014 RNSM and SC 156 Terminal Authentication Success event

13 Call Control 008 RNSM and SC 150 Session Configuration Failed Event


5-info Call Control N/A RNSM and SC 164 Session Configuration Failed

13 Call Control 008 RNSM and SC 157 Terminal Authentication Failure

5-info Call Control N/A RNSM and SC 174 Terminal Authentication Failure

5-info Call Control N/A RNSM and SC 167 Closing Session – Unknown UATI

5-info Call Control N/A RNSM and SC 414 Session did not Receive HW ID Resp. from the AT

Sample RNC Log Event Outputs:

Session State Transitions

SCSM Event - (Attempt)

Session Configuration Success

Terminal Authentication Success

Session Configuration Failed Event

Session Configuration Failed Event

Terminal Authentication Failed


Session Failed – (Unknown UATI)

Session Failed – (No Hardware ID Response)

Sample from RNC Logs (Session Config Failed):

SSM SSMWaitForConfigDone: RX from SCM: Session Configuration Failed Event SSM SSMWaitForConfigDone: Closing Session – Session Configuration Failed SCSM SCSM_ANInitiated: Received Deactivate Event SCSM: Transitioning from SCSM AN_Initiated to SCSM SCSM_Inactive CSM CSMOpen: Rx: Local Connection Close Event Reason 2 CSM CSMOpen: Tx to AT: Connection Close Event Reason 2 CSM CSMOpen: Exiting State CSM CSMAwaitATClose: Entering State Session: Transitioned from SSMWaitForConfigDone state to

SSMWaitForAppsToDisable state SCSM: Transitioning from SCSM SCSM_Inactive to SCSM SCSM Inactive SCSM SCSM Inactive: Processing ProtocolSubtypeConfig Configuration Response Timeout SCSM SCSM Inactive: Received invalid Configuration Done Indication Event

Note: SSM = Session State Machine, CSM = Connection State Machine, and SCSM = Session Configuration State Machine

A13 Dormant Handoff Sessions With release 3.0, the A13 dormant session handoff feature is added. This feature forces a session handoff for dormant ATs based on the following: • UATI request with foreign UATI (UATI from session on source RNC) • UATI request with RATI (prior session handoff, usually occurs when the AT has lost supervision

prior to A13 handoff request In most cases A13 handoff should be successful. The following are some typical failures that may be a result of the process not completing: Possible A13 Session Failures • Target RNC fails to locate the source. This typically means that the source is not configured in the

target’s A13 configuration table • Session does not exist at the source RNC and consequently the source sends an A13 reject. • Configuration parameters are not acceptable to the target. This may happen if there is some delta

in session configuration between the RNCs • Failure can also occur in some fast inter-RNC ping pong cases.


The following RNC log sample shows an A13 (foreign UATI) handoff as seen from the target RNC:

A13 Session Ping Pong Scenarios The purpose of this section is to discuss potential performance issues related to fast ping-pong at the RNC boundaries. By fast ping-pong, we assume that AT switches between subnets faster than the speed of session transfer process itself. Here we will analyze different scenarios and point out the problems and impact on performance.

CASE 1: Target (b) receives A13 Response from the source (a), but AT ping-pongs back to the source (a), before UATI is assigned.

In this case there will be no problem since when AT bounces back to the source (a) RNC it will use its old UATI. Target (b) will assign new UATI, so there will be temporary waste of session memory at the target (b), but this will clear after a time-out. Target (b) will not send A-13 confirm to the source (a).

CASE 2: Target (b) receives A13 Response from the source (a), assigns new UATI, but AT ping-pongs back to the source (a), before UATI assignment procedure is completed.

In this case AT may have received new UATI from the target (b), but the target does not know this. So when AT bounces back to the source (a) it will attempt new dormant handoff using UATI it got from (b). When (a) sends A-13 request to (b) it will be rejected since there is no record of this session at (b). Upon receiving A13-Reject, (a) will close session and proceed with new session establishment.

Therefore the consequence of this scenario is that session is closed and needs to be re-established. This is slow and it does require set-up of traffic channel. Also, in this scenario numDormantHandoffAttempts is pegged, but not numDormantHandoffSuccesses. Therefore this constitutes session setup failure.

Such fast ping-pong scenarios can occur in challenging RF situations, with no clear dominant server, where UATI procedures take long to complete.

CASE 3: Target (b) receives A13 Response from the source (a), and UATI complete from the AT, but AT ping-pongs back to (a) while re-configuration of session protocols is under way.

Same as in case 3, when AT bounces back to (a) RNC will send A-13 request to (b), which will be rejected.


Upon receiving A13-Reject, (a) will close session and proceed with new session establishment.

Therefore the consequence of this scenario is that session is closed and needs to be re-established. This is slow and it does require set-up of traffic channel. Also, in this scenario numDormantHandoffAttempts is pegged, but not numDormantHandoffSuccesses. Therefore this constitutes session setup failure.

Such fast ping-pong scenarios can occur in challenging RF situations, with no clear dominant server, where UATI procedures take long to complete.

CASE 4: Target (b) receives A13 Response from the source (a), and UATI complete from the AT; completes the re-configuration of session protocols, but AT ping-pongs back to (a) before Hardware Id exchange is completed.

Consequence would be the same as in cases 2 and 3, but this should not occur frequently, because A-13 should be configured to pass hardware id in A-13 Response message and therefore hardware id exchange is skipped at the target (b).

CASE 5: Target (b) receives A13 Response from the source (a), and UATI complete from the AT; completes the re-configuration of session protocols and hardware Id is retrieved, but AT ping-pongs back to (a) before (b) receives Location Notification.

In this case A10 between (b) and PDSN is not opened. Also A13 Confirm is not sent to (a), so old session remains opened at (a). When AT bounces back to (a), that RNC will request session info from (b). This will complete and (a) will open new session. However at hardware id stage (a) will realize that it already ahs [old] session for that AT and will proceed closing it. This closes Associated A10 as well, which in turn clears PPP at the PDSN. Note that AT still keeps the PPP session so there is a state mismatch.

10.1.1.3 OMs

The following OMs should be observed for Session Setup Performance:

Regular Session Setup Failures: 1xEV-DO Session Setup Failures = totalSessionSetupsFailed where: totalSessionSetupsFailed = numSessionSetupsFailedToUATICmpltTimeout+ numSessionSetupsFailedToInvalidUATICmpltMsgSeq + numSessionSetupsFailedToSessionConfig + numSessionSetupsFailedToTermAuth + numSessionsTerminatedToHwIdRspFailure + numSessionSetupsFailedToInvalidHwIdValue + numSessionSetupsFailedToInvalidHwIdType + numSessionsTerminatedToAtIdRspTimeout + numSessionSetupsFailedToATInitiatedSessionClose + numSessionSetupsFailedToRNCInitiatedSessionClose+ numSessionSetupsFailedToSessionInfoConfirm + numSessionSetupsFailedToOtherCauses numSessionSetupsFailedToSessionConfig - Total number of Session Setups which failed on this DO-RNC due to the Session Configuration failure. It includes the scenario when the connection was failed or dropped during session setup. numSessionSetupsFailedToTermAuth - Total number of Session Setups which failed on this DO-RNC due to Terminal Authentication failure.


numSessionsTerminatedToUnknownLocalUati – Total number of sessions which were terminated by the DO-RNC due to the received local UATI being unknown. It happens when the color code in the UATI matches the color code of the DO-RNC but the session instance does not exist on the DO-RNC. This attribute is pegged when any message with an unknown local UATI is received, not just the UATI Request message. This is a state mismatch when the AT thinks that it has a 1xEV-DO session with the AN, but the AN does not have a session with the AT. numSessionsTerminatedToHwIdRspFailure - Total number of sessions which terminated on this DO-RNC due to Hardware Id Response Failure. Note that it is a session setup failure, not a session termination. Session setup is not completed before the HardwareIdResponse message is received. numSessionsTerminatedToAtIdRspTimeout - Total number of sessions which terminated on this DO-RNC due to AT Id Response Timeout. Note that it is a session setup failure, not a session termination. Session setup is not completed before receiving AT Id Response with a result of “OK”. The AT Id Response is an internal message from the SC card. It is not a standard airlink message. This attribute IS supported. numSessionSetupsFailedToOtherCauses - Total number of Session Setups which failed on this DO-RNC due to other causes such as the DO-RNC not receiving UATI Complete or UATI Request message from the AT. A13 Dormant Handoff Failure OMs: UATI Request initiated A13 handoff OMs Percentage of Regular A13-Dormant Handoff Failures due to A13- REJECT from the Source RNC = 100% * numA13TotalRejectRNC / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) Percentage of Regular A13-Dormant Handoff Failures due to Possible Source RNC Table Datafill Error = 100% * (numDormantHandoffFailureToSourceLookupFailureRNC + numDormantHandoffFailureUati104RNC) / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) The following could be caused by a connectivity problem between the RNCs. If A13 is not enabled on the source RNC, it can also cause this failure. Percentage of Regular A13-Dormant Handoff Failures due to A13 Message Timeout = 100% * numA13ReqTimeoutRNC / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) Percentage of Regular A13-Dormant Handoff Failures due to Socket Issue = 100% * numDormantHandoffFailureSourceUnreachableRNC / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) Percentage of Regular A13-Dormant Handoff Failures during the


UATIAssignment Stage = 100% * (numDormantHandoffFailureNoUatiReqRNC + numDormantHandoffFailureNoUatiCmpltRNC + numDormantHandoffFailureInvalidUatiCmpltRNC + numDormantHandoffFailureNoRncResourceRNC) / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) One possible cause of the following failure is the different setting for terminal authentication between the source and target RNCs. Percentage of Regular A13-Dormant Handoff Failures due to Retrieved Session Configuration Unacceptable = 100% * numDormantHandoffFailureRetrievedConfigUnacceptableRNC / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) The session configuration failure is usually caused by the connection setup failure or connection drop during the session configuration phase. Such events are usually due to the poor RF condition along the DO-RNC subnet border. Percentage of Regular A13-Dormant Handoff Failures during the Session Configuration Stage = 100% * numDormantHandoffFailureSessionConfigDuringReconfiguration RNC / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) Percentage of Regular A13-Dormant Handoff Failures during the Terminal Authentication Stage = 100% * numDormantHandoffFailureTAAfterA13RspRNC/ (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) The event pegged by numDormantHandoffFailureHdwIdTimeoutRNC is frequently caused by poor RF condition. The airlink messages associated with hardware ID retrieval could be lost in many cases. Percentage of Regular A13-Dormant Handoff Failure during the Hardware ID Acquisition Stage = 100% * (numDormantHandoffFailureHdwIdTimeoutRNC+ numDormantHandoffFailureInvalidHdwIdValueRNC+ numDormantHandoffFailureInvalidHdwIdTypeRNC) / (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) The AT ID validation happens within the DO-RNC. Such failure is caused by the DO-RNC and not related to any external environment. Percentage of Regular A13-Dormant Handoff Failures during the AT ID Validation Stage = 100% *


numDormantHandoffFailureAtIdTimeoutRNC/ (numA13TotalRejectRNC + numTotalDormantHandoffFailureRNC) Prior Session Initiated Handoff OMs Similar OMs as those given for regular A13 initiated dormant handoff can be developed for prior session initiated dormant handoffs. For more information on A13 handoff see reference [21]. Using OMs to diagnose common problems (See [21])

OM Common Diagnosis Scenario Attempts Number of times a dormant handoff was attempted Successes When the number of successes is close to the number of attempts

the system is operating in a healthy manner. Rejection Messages Increments because either, the wrong source RNC was chosen, or

else the AT did not complete setting up its session on the RNC prior to moving to another RNC. However, if there are a large number of rejections due to “Session not Found” then another possibility is that the subnet ID was misconfigured in the Source RNC lookup table.

Failure – Source RNC not Found A13 is disabled at the target RNC At attempted a dormant handoff from a Source RNC not listed in the Source RNC lookup table.

Failure No A13 Response Either the source RNC is out of service or there is a connectivity problem. Try to ping the source RNC. The following may be reasons for no A13 response from the Source RNC: • A13 is disabled on the source RNC, • The IP address on the source RNC was incorrectly configured in

the target RNC’s “Source RNC lookup table. • Source RNC is down • IP connectivity issue.

Failure – Session Configuration Unacceptable

Error while parsing configuration attributes received within the A13 response. Could indicate that either, the message was not formed properly, or else an error occurred in the transmission. Error while trying to reconfigure the session parameters at the target after a successful A13 retrieval. This could occur to the airlink connection being lost.

Failure – Hardware ID After a successful A13 retrieval at the target, there was no hardwareID response during session setup. This could occur due to the airlink connection being lost or the AT moved away before sending the response, and does not necessarily indicate any problem.

Failure – Missing UATI Complete

After a successful A13 retrieval at the target, there was no UATIComplete response when assigning a new UATI. This could occur due to the airlink connection being lost or the AT moved away before sending the response, and does not necessarily indicate any problem.

Failure – UATI -104 Mismatch After a successful A13 retrieval at the target, it was determined that the AT UATI-104 did not match the source RNC. The most likely reason is that the wrong source RNC was selected (due to colorcode reuse). This does not indicate any


operational problem. Failure – UATI-104 Matches Local Subnet

This is a very rare error that can occur during a prior session handoff. This requires the AT to ping-pong between subnets rapidly. If the AT attempts a prior session on the target and if the prior session attribute matches the local subnet, then this error is pegged.

Failure – Source RNC unreachable

A socket problem occurred on the target RNC. If this counter constantly increases, it could indicate a problem with the system. Determine which slot has the problem (use the per-slot OMs) and reboot that slot.

Failure – Missing UATI Request UATI Request was not received after an initial foreign UATI message was received at the target. This may be due to the AT moving away after the initial message. This does not indicate an operation problem.

Failure – internal errors This may indicate that the RNC is either temporarily overloaded, or has encountered internal resource problems. If this OM continues to increment over a long period of time, it may be prudent to reboot the RNC.

Post Dormant Handoff reconfiguration needed

Indicates that the source and target RNC has different operator settings. Does not necessarily indicate any operational problems.

A13 Message Count – To Remote RNCs

If this counter does not increment during dormant handoffs, then A13 handoffs might be disabled.

A13 Message Count – From Remote RNCs

If this counter does not increment during dormant handoffs then A13 might be disabled or there may be a connectivity problem.

10.1.2 KPI Calculation 10.1.2.1 KPI Value and Tolerance Factor

Session Setups should be a rare event in that Sessions are long-lived. The Session Setup Success rate shall be > 95% and should be > 98%. Because of the current design of the product the success rate is ~ 50% as measured via OMs. There is no A13 support and each Subnet Id is identified to a single RNC. This creates many borders in a normal network causing a large number of session setup attempts in these border areas. Because these are border areas, the RF channel is by definition less than optimal, increasing the failure rate.


10.1.2.2 Calculating KPI from AT Logs

Session Setup Success Rate = Session Setup Successes / Session Setup Attempts = # of UATI Request Messages / # of Hardware ID Response Messages

Example From AT logs:

10.1.2.3 Calculating KPI from RNC Logs After downloading the RNC logs, use a macro to extract all “Transitioned from SSMWaitForAtIdRsp state to SSMWaitForLocationNotify state” and “Transitioned from SSMWaitForUATIReq state to SSMWaitForUATIRspFromRC state”. Use the following formula to calculate the Session Setup Success Rate:

Formula: Session Setup Success / Session Setup Attempts = Transitioned from SSMWaitForAtIdRsp state to SSMWaitForLocationNotify state / Transitioned from SSMWaitForUATIReq state to SSMWaitForUATIRspFromRC state

Currently NEWTUN does not provide specific statistics related to A13 success. However, the success rate can be determined by:

= RATIIForeignUATwquestsUATI

ConfirmsARNCSuccessA //_Re#13#%13

10.1.2.4 Calculating KPI from OMs

The OMs required to determine the session setup and A13 handoff success rate are calculated as follows:

Regular EVDO Session Setup Success Rate 1xEV-DO Session Setup Successes = numSessionSetupSuccessful Therefore, Regular 1xEV-DO Session Setup Success Rate = numSessionSetupSuccessful / (numSessionSetupAttempts – numSessionSetupsFailedToATInitiatedSessionClose – numSessionSetupsFailedToRNCInitiatedSessionClose) UATI Request Initiated Regular A13 Dormant Handoff Success Rate


The UATIRequest Initiated Dormant Handoff (also known as Regular A13-dormant handoff) is triggered when an AT crosses the subnet border without losing supervision of the 1xEV-DO carrier. However in order for the A13-dormant handoff to proceed, the A13-dormant handoff must be enabled on the target RNC. Regular A13-Dormant Handoff Success Rate = Regular A13-Dormant Handoff Successes / Valid Regular A13-Dormant Handoff Attempts Where: Valid Regular A13-Dormant Handoff Attempts = (numDormantHandoffAttemptsRNC – numDormantHandoffFailureATInitiatedCloseRNC – numDormantHandoffFailureRNCInitiatedCloseRNC)

Prior Session Initiated Dormant Handoff Performance A prior session initiated dormant handoff is triggered when the AT, during the session configuration phase, requests to retrieve a prior session from the source RNC. When such request is received, the target RNC will attempt to retrieve the session information from the source RNC via the A13 interface. For the prior session retrieval to proceed, the A13-dormant handoff must be enabled on the target RNC. Prior Session Initiated Dormant Handoff Success Rate = numDormantHandoffSuccessesPriorSessionRNC / (numDormantHandoffAttemptsPriorSessionRNC – numDormantHandoffFailureATInitiatedClosePriorSessionRNC – numDormantHandoffFailureRNCInitiatedClosePriorSessionRNC) Nortel- XX# show 1xEVDO counters all

The following equation should be used to determine the Session Setup Success Rate from OMs:

Formula: numSessionSetupSuccessful/(numSessionSetupAttempts-numSessionsTerminated ToReceivingUatiReq )

10.1.3 Troubleshooting Guide DOM Health First, perform basic readiness check-up on the DOM with session setup failures (see Appendix A). Log on to the DOM and check: • Are the modules up and running? • Is DOM node up? • Are sector-elements up? • Is Abis Peer up? • Is there an IP connectivity to the RNC ? • Check the GPS timing source • Check the SNTP Time from GPS • Check if there are traffic channels between the DOM and RNC • Check if there are sessions and connections in this DOM Events related to Session Setup Failure: 1. Repeat Offenders: Often Session setup success rate can be adversely impacted by stationary terminals close to an RNC border that periodically tune to a sector that is on the adjacent RNC. When these periodic


session setup events occur between RNCs, the failures that result skew the overall network performance trend although the impact as a whole is only to a handful of users. • If the AT does not complete Terminal Authentication or reach the Hardware ID retrieval phase of session

setup IMSI information is unavailable to identify the user. • A method for troubleshooting Repeat offenders will be presented in a later release of this document.

2. Session Configuration Failure: Look for “Session Configuration Failed Events” in the RNC logs. From the OMs look for “numSessionSetupFailedtoSessionConfig”

3. Terminal Authentication Failure: Look for “TerminalAuthenticationFailureEvents” in the RNC logs. In the OM’s look for “numSessionSetupsFailedtoTermAuth”.

4. Follow the following steps to troubleshoot Terminal Authentication or Session Configuration problems:

a. For terminal authentication to work the AT, DO-RNC, and AN-AAA servers all should be configured properly.

b. Required AT Configuration:

i. NAI – must be equal to the username configured for the AT in the AAA server(s) ii. Password - must equal the password setup for the AT in the AAA server(s).

iii. Compare the AT’s Network Access Identifier (NAI) and Password to the Username and Password configured for the AT in the AAA server, as follows: a. On the AT, use QPST (or another provisioning tool) to display the NAI and Password for

the AT, typically found on the 1xEV tab. b. On the access network AAA server, check the user profile for this AT and display the

username and the password for the particular user. If the AAA’s User name for the AT does not match the NAI setting configured in the AT, or if the AAA’s Password for the AT does not match the Password configured in the AT, there is a misconfiguration of the authentication parameters for this AT. To recover, use QPST to configure the AT’s:

• NAI to equal the Username configured for the AT in the AAA server • Password to equal the Password configured for the AT in the AAA server

c. Required DO-RNC configurations: i. IP address of the AAA server

ii. Display the IP address the DO-RNC uses for A12 terminal authentication communications with the AAA server with the Enable mode show termauth configuration command, as follows:

NORTEL-XX#show termauth configuration =============================== TA Scalar Configuration Record =============================== Terminal Authentication : Disabled A12 Interface Source IpAddress : 10.12.0.241 A12 Maximum Retransmit Attempts : 6 A12 Access Timeout Value : 1 A12 Load Balancing Mode : NAI realm based Aaa Routing Table Max Size : 64 Aaa Routing Table Default Route : 1 Default AnPPP Auth Method : CHAP Default ACCM value : 0x00000000 AnPpp IdleTimeout value : 180 ===============================

Ensure that the A12 Interface Source IpAddress is the IP address the AAA server(s) uses to communicate with DO-RNC clients. If the IP address is incorrect, configure the AAA server with the A12 Interface Source.

iii. Ensure the secret key that the AAA server and the DO RNC use to validate A12 communications


iv. Examine and verify the Authport for each AAA server listed.

The Authport is the IP port number on which the AAA server is configured to listen for terminal authentication requests. This is typically 1812. If the AAA is configured to listen on a different IP port number, modify the DO-RNC configuration of the AAA to match the correct port number with the Configure Termauth Server mode server <name> <secret> <A.B.C.D> [authorization port number] command, as follows, assuming a name of AAA1, a secret of xyz, and a AAA server IP address of 10.11.12.13, and an authorization port number of 1812: NORTEL-XX(config-termauth-server)#server AAA1 xyz 10.11.12.13 1812

vi. Ping each listed AAA server IP address from the DO-RNC with the Enable mode ping

<Ip address> command, as follows:

NORTEL-XX#ping <AAA server ip address> If the ping fails, check the AAA’s actual server IP address. If it is not the same as configured in the DO-RNC, change the DO-RNC’s configuration with the Configure Termauth Server mode server <name> <secret> <A.B.C.D> command, as follows, assuming a name of AAA1, a secret of xyz, and an AAA server IP address of 10.11.12.13: NORTEL-XX(config-termauth-server)#server AAA1 xyz 10.11.12.13

vii. To check current authentication activity use the following command while in Enable mode at the

RNC: Nortel-XX# show termauth statistics scalar Useful OMs used to diagnose A13 HO problems


10.2 Session Setup Time



Same as for session setup success rate.

10.2.1.2 RNC Logs There are currently no good means for measuring session setup time from the RNC logs but may be added in future releases of NEWTUN.

10.2.1.3 OMs There are no system OM’s currently available that track session setup time. However, the engineer can go through CLI to view the Max, Min, and Ave Session setup time.

Nortel-XX# >show 1xEVDO counters all

Session Setup Time :-

Minimum : 00:00:02.320

Maximum : 00:00:02.320

Average : 00:00:02.320


From [14]: EVDO <= 8s EVDO + PPP <= 14 s

Because this should be a rare event, we can afford more time for a setup without end-user performance degradation. But since Session Setup is a normal system event using normal system messages and signaling, the time for setup should be defined. Session Setup is affected by the amount of configuration data that must be negotiated. Session Setup Time: 11-12 seconds for unloaded network and 14-15 seconds for loaded network (with TA). {We need to define start/stop points and test method. This will have to be measured with drive testing}


EV-DO session setup time is measured from UATI Request message to Hardware ID Response message. In RFO, open Statistics Tool Session Statistics EVDO Sessions. The statistics can be exported to Excel.


Example: RFO Statistics View EVDO Session Statistics

10.2.2.3 Calculating KPI from RNC Logs 10.2.2.4 Calculating KPI from OMs

10.2.3 Troubleshooting Guide

10.3 Connection Setup Time


Method of Collection: 1. Configure Autocall in Optis-M or XCAL-DO to perform an FTP download of a 100kByte file with a

15second interval between data calls.

2. Insure that the AT is in HDR only mode and receiver diversity is enabled prior to beginning the test.

3. Insure that all EVDO log mask attributes are selected.

4. Perform a drive test over the cluster golden route to collect AT logs

5. At the end of each cluster drive stop logs and save for post processing.

10.3.1.2 RNC Logs RNC logs can be collected simultaneously with the mobile logs for the purpose of measuring connection setup time. However, the present set of RNC log analysis tools does not calculate connection setup time, therefore the engineer will have to develop scripts that will filter out the AT UATI and calculate the time deltas between Connection Request and Traffic Channel Complete.

10.3.1.3 OMs Method of Collection:

1. There are currently no specific OMs for measuring connection setup time per UATI or overall.

2. Per UATI connection setup statistics can be collected from the CLI prompt but the statistics are only for the last or current AT connection.

NORTEL-XX# show 1xEVDO connection <UATI> counters all Connection Setup Time : 00:00:00.220


3. The average connection setup time for the entire network can be determined from CLI using the following command:

NORTEL-XX# show 1xEVDO counters all Connection Setup Time:- Maximum: 00:00:00.280 Minimum: 00:00:00.150 Average: 00:00:00.220

4. At the end of the cluster drive test log into the CLI prompt


From [14]: 2s for 95% of events (AT initiated)


Please see section 6.2.4. Example: RFO Connection Statistics

10.3.2.3 Calculating KPI from RNC Logs Will have to be done manually at this time

10.3.2.4 Calculating KPI from OMs As shown is section 7.3.1.3



10.4 Access Failures


Method of Collection: Configure the Autocall script in Optis or Invex3G for download of a 500 KB test file with the idle / wait time set for 15s. Configure the number of calls for 1000 calls such that a significant sample of data can be collected. (See Below):

Perform a drive test of the cluster Golden Route.

FTP downloads of 100kB file Post Process AT logs with RFO

From Connection Statistics in RFO, is the average connection time

exceed 2 seconds END

N

From the Access Attempt Message is the Probe Sequence Count greater

than 1?

Y

Identify top ten sectors with longest connection setup times. Is ROT

excessive? NORTEL-XX#show rate-control rev-link-stats <sector> e1=alpha

- Reduce the Num Steps for Probes to 4. - Increase the Power Step Size for Probes to 12 (6dB) per probe. - Increase the Probe Initial Adjust from 0 to 6dB. - Verify the Access Cycle Duration = 64 slots - Verify new settings from Access Parameters

Y

N

Y


Example: Optis-M AutoCall Setup for Access Failures


Example: Invex3G Data Connection Task Setup for Access Failures

1. In Optis-M Insure that the “Allow Dormant State” checkbox is not checked unless it is desirable to have the AT wait for the data inactivity timer to expire such that the system closes the RF connection between downloads.

2. For data collection select all EVDO attributes for the log mask

3. Drive the golden route to collect AT logs. 10.4.1.2 RNC Logs 1. Log into the DO EMS and within the Log Facility manager configure the SC cards to collect RNC logs at severity level 16. Logging can also be enabled on the RNSM cards individually as given in section 5.3.1 for setting up RNC logging

2. RNC log events associated with Call Setup (AF) Failures are shown in the table below:

RNC State RNC Log Message CSM CSMAwaitCloseSHOL

TX to AT: Connection Deny Message (This message indicates that resources were unavailable to support call setup at this time. Usually associated with a state mismatch in the RNC)

SSM SSMAwaitforConfigDone

Closing Session – Unexpected UATI Request (This message indicates that the RNC was waiting for the AT to respond to a previous message from the AN)

CSM CSMAwaitTCC Traffic Channel Complete Message not Received from AT in time (This describes an exception in the Connection State Machine indicating the RNC (while in the process of opening a connection)


did not receive the TCC in time) CSM CSMAwaitTCC No pilots in active set at time of sending

Traffic Channel Assignment message (Exception message from the Connection State Machine indicating that the RNC had a problem sending out the TCA to the AT.)

10.4.1.3 OMs 1. In order to setup data collection, follow the guidelines in section 5.4.4 to setup the data collection

configuration with the RNCPerfBySectorCarrier_R3.0 template selected.

2. The following OM’s are available from the RNCIS856 Sector Carrier Performance Table:

numATConnectionSetupsFailedTccTimeoutSC numANConnectionSetupsFailedTccTimeoutSC numFCConnectionSetupsFailedTccTimeoutSC o The overall per RNC FCA rate is given by the following level 1 formula:

Percentage of Overall Connection Setup Failure rate = ((numConnectionSetupsBlockedByRncResources + umConnectionSetupsFailedByRncResources + numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted) / (numConnectionRequestsFromAT + numFastConnectsAttempted – numConnReqsWhileSettingUp – numConnReqsWhileTearingDown – numConnReqsWhileOpen)) * 100

3. The following level 2 and 3 OMs show the percentage of connection setup failures broken out according to the following categories:

a. Unsuccessful DO-RNC resource allocations – numConnectionSetupsBlocked +

FailedBy_[RNCResources / RN] o Includes events such related to RNC and DOM resources

b. RF Failures – numConnSetupsFailedRU + TCCTimeout o RF failure metrics include events related to TCC timeouts, and RU timeouts etc, that

contribute to the overall connection setup failure rate. c. Misc Failures – cnumConnSetupsFailedSWError

o Usually includes events like software errors

o FCA OMs are available for Session related FCA and are denoted by (sNum***) whereas regular data oriented connection setup failures are denoted by (cNum***).

10.4.2 KPI Calculation 10.4.2.1 KPI Value and Tolerance Factor From [12] Access Failure% <= 4%?

10.4.2.2 Calculating KPI from AT Logs 1. Post process the AT logs with RFO 2. Open the Call Statistics window in RFO and look at the number of Access Attempts and Access

Failures.


Example: EVDO RFO Statistics Window

3. The Access Failure Rate from the AT logs is defined as follows: For HDR only data collection = DO Access Failure Rate = DO Access Failures / DO Attempts For Hybrid data collection = Access Failure Rate = {(DO Access Failures + 1x Access Failures)/ (Do Attempts + 1x Originations + 1x Page Reponses)}

10.4.2.3 Calculating KPI from RNC Logs 1. To determine the number of access failures from RNC logs the use of the NEWTUN post processing

tool is recommended. 2. From NEWTUN, open the database containing the post processed RNC log files. 3. The default screen is the statistics view. From the statistics view select Access Terminals All. 4. Sort the list by IMSI. 5. Look for Total Access Failure Rate in the statistics.

10.4.2.4 Calculating KPI from OMs 1. To determine the access failure rate from the OM’s the following: equation should be used: 2. Total Failed Connection Attempts can be broken down between session connection setup attempts and

data connection setup attempts. OMs are available to collect both cases. Session OMs are prefixed with an [sNum] and data connection attempt / failure OMs are prefixed by [cNum]

Failed Connection Attempts (FCA) = (cNumRFRelatedFCA + cNumResourceRelatedFCA + cNumMiscFCA) / cNumConnectionSetupAttempts

3. Unsuccessful DO-RNC resource allocations

Percentage of Connection Setup Failures due to Unsuccessful DO-RNC Resources Allocation = 100% * (numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRncResources) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout +


numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted)

4. Unsuccessful DOM Resources allocations

Percentage of Connection Setup Failures due to Unsuccessful DOM Resources Allocation = 100% * (numConnectionSetupsBlockedByRn + numConnectionSetupsFailedByRn) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted)

5. Connection Setup Failures due to RF Failures Percentage of Connection Setup Failures due to RF Failures = 100% * (numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted) 6. Connection Setup failures due to Misc. Reasons Percentage of Connection Setup Failures due to Misc Failures = 100% * (numConnSetupsFailedRncTimeout + numConnSetupsFailedSWError + numConnSetupsAborted) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted


10.4.3 Troubleshooting Guide Call Failure Reasons from RNC logs From the NEWTUN Statistics view, select Network Statistics. From the network statistics it is possible to determine the DOM(s) associated with each access failure.

1. More detailed information regarding each access failure can be determined by selecting the Connection Details tab from the statistics view and selecting the Connections List.

2. Form the Connections List sort by Failure Classification Access Failure. This will pull up all the access failures and the connection / RF attributes seen just prior to the failure.

3. If a failure reason cannot be readily inferred from the data in the connections list, click in the access failures in the Network Statistics which will add the connection attempts that ended in a setup failure to the Active List.

4. Select the call in question from the Active list to view the call sequence in the Message View.


NEWTUN Message View

5. Information about the distance the AT was from the site can be obtained from the Cell Radius Tool in

NEWTUN. 6. By matching the DOM IP and sector PN where the call setup failed from the Network Statistics view

in the Cell Radius Tool the distance and relative Ec/Io of the originating PN can be determined.

New CR

Multiple TCA’s without an ACK from the AT.


NEWTUN Cell Radius Tool

Operational Measurements 7. Operational measurements can be used further to determine the top contributors to connection setup

failures. In the case of Failed Connection attempts, these are typically considered RF or equipment error related o Percentage of Connection setup failures due to RF Failures (Level 2, per RNC) o This level 2 metric includes events which are related to TCC timeouts and RU Timeouts and

contribute to the overall connection setup failures. o When this metric evaluated with other level 2 metrics, it can be easily determined that which

level 2 metric is HIGHER contributor to the overall connection setup failures. 8. Percentage Contribution of TCC Timeouts to Connection setup failures due to RF Failures (Level 3,

per RNC) o If this metric is the highest contributor among level 3 metrics, it indicates the issues related to

TCC timeouts. o TCC timeout happens when Call Control on RNC times out waiting for Traffic Channel

Completes message from the AT. o Higher TCC timeouts indicates that the RF conditions were not good enough for the TCC

message to reach the DOM or TCA message reach the AT or TCC Wait Timer parameter value is not good enough. Link budget need to be verified for the forward and reverse link coverage balance. Power used (configurable parameter) for sending TCA need to be verified

Calls originate far (>3.4km) from the site. This could be an overshooting site.


9. Percentage Contribution of Route Update Timeouts to Connection setup failures due to RF Failures (Level 3, per RNC) o If this metric is the highest contributor among level 3 metrics, it indicates the issues related to RU

timeouts. o RU timeout happens when Route Update message is not received by the RNC from AT within a

predefined time interval. The timer used for this event is not configurable. o Higher RU timeouts indicates that the RF conditions were not good enough for the RU message

to reach the DOM. Other contributing factors to this event exists but could not be fixed by us (needs to have design take care of the software). Link budget need to be verified for the forward and reverse link coverage balance

Verification of System Readiness • Are the modules (DO Network Elements) up and running?

o Module Status in EMS must be “Active” NORTEL-XX>show module o If not all of the NEs are physically present or do not have an Active status see “NE Outage

Troubleshooting Flowchart” on page 3-10 of NTP 411-2133-949. • Insure all Interfaces are operational (i.e. T1/E1, PPP interface for each deployed T1, node 1/0/1

interface). o From CLI use the following commands to check the interface: NORTEL-XX#show

interface o If any of the interfaces have an operational status of “Down” proceed according to the IP RAN

troubleshooting flow chart in section 3-11 of NTP 411-2133-949: • Is DOM node up?

o From the RNC use NORTEL-XX#show node to verify the operational state of the DOMs

o If the operational state is “Down” follow the troubleshooting flowchart on page 3-10 of NTP 411-2133-949

• Are sector-elements up? o On the DOM check the administrative and operational status of sector elements NORTEL-

XX#show sector-element o If the operational status is “down” then there is a problem with Abis or topology manager o Follow recovery procedure 4-16 in NTP 411-2133-949 to re enable Abis or topology manager.

• Is Abis Peer up? o ABIS is a TCP protocol used to carrier user signaling and data traffic between the DOM and the

DO-RNC o IF the ABIS peer is down it is common for the ATs to fail session setup and connection setups. o To verify connectivity to the ABIS peer, Ping the RNSM module from the DO-RNC

NORTEL-XX>ping 127.1.(RNSM slot).1 o Another way to verify connectivity is to check if there is an ABIS peer by performing the

following CLI command: NORTEL-XX#show abis peer. If there are no active connections listed there could be a problem with the Topology manager or Abis. (See NTP 411-2133-949 for more info)

• Is there an IP connectivity to the RNC ? o A route to every DOM PPP interface IP address, and routes to PDSNs, the DO- EMS server, and

the AAA server. o A standard Ping test can be used to determine if any of the required routes between a source and /

or destination route is missing. o Use recovery procedure 4-14 - 15 in NTP 411-2133-949 to configure missing routes between NEs.

• Check the GPS timing source o Use the following to check for the presence and lock status of DOM GPS NORTEL-

XX>show gps health o If GPS health is not correct check GPS hardware

• Check the SNTP Time from GPS o All network elements require a timing source GPS etc.


o For DO-RNC’s the timing source must be the IP address of a DOM serving as an SNTP server. For DOMs the timing source must be GPS

o Use the following command to show the SNTP time NORTEL-XX>show sntp time o If the NE is the DO-RNC and the SNTP time is incorrect see “TroubleshootDO-RNC Time

Reference” on page 4-31 in NTP 411-2133-949 • Check if there are traffic channels between the DOM and RNC

o Check for Abis Peer NORTEL-XX#show abis peer • Check if there are sessions and connections in this DOM

o Check for Sector Carrier connections NORTEL-XX#show 1xevdo rn 10.10.10.10 counters

o Verify that “Airlink Resources Currently Allocated” is not equal to zero. Other Events Associated with DO Access Failures • To accurately see where problems occur that result in Access Failures it is necessary to collect and

correlate AT and RNC logs. • RNC Time Stamps Note: It should be noted that the RNC log time stamps are given in GMT time,

therefore the timestamps will have to be adjusted for the local time zone in order to correlate the RNC and AT logs.

• Access Fails following Connection Drops – Troubleshoot connection drop symptoms first. • Access Fails due to insufficient cell coverage radius – Data connection access failures due to ATs

being in the fringe of coverage. • Coverage radius statistics can be determined using the Cell Radius Tool in NEWTUN.

• In the event that per sector carrier access failures are occurring due to poor RF coverage at an average distance, steps should be taken to deny access to the site beyond the pre determined coverage radius. o Collect RNC logs with severity set to 16. o The distance from the serving cell can be determined by reviewing the last “Route Update Detail”

prior to an Access Failure. o Within the Route Update Detail there is a parameter called “EPNO” that gives the RTD value

which can be used to determine the distance from the site prior to failure.

Poor C/I and high TX Power


o Calculate the average RTD for all Access Failures on a per DOM basis. This value will be used to set the Cell Radii for Cells with a high AF rate.

o To modify a carrier associated with a particular DOM open the DOEMS and go to the DOM to be modified by opening the “Network Database” in the left hand pane under DOEMS.

o Expand the “Network Database” and select “Carriers”. o In the Right-hand pane select the appropriate DOM according to IP address.

o From the IS856 Carrier Menu select “Modify Configuration” o Change the “Cell Radius” parameter to the proper number of chips to limit access to the sector

from outside a specific radii. o Where (Meters = # Chips x 244m/chip) to specify access radius

o Once the table has been modified, Click Submit to save the changes.

10.5 Call Blocks

10.5.1 Data Collection Methods 10.5.1.1 AT Logs Method of Collection:

1. Tests of Call Blocking rate should be performed in a loaded network under busy hour conditions and should generally encompass all users on the system.

2. To get average per user Blocking statistics, configure autocall in Optis-M or the data connection tab in Invex3G to perform FTP downloads of a 100KB test file (small files so number of connection attempts will be statistically significant). Insure that the idle / Wait timers in the data collection setup are set for 15seconds.

3. Select all EVDO attributes in the log mask.

4. Drive test the cluster to collect the AT logs.

10.5.1.2 RNC Logs Method of Collection:

1. Go to the DO-EMS and configure the log facility manager to collect Control Logs as follows:

Severity Level

Component ID DO-RNC CPUs Message ID

Description

13 Call Control 009 RNSM and SC 231 No RC Resources to Allocate


2. RNC logs should be collected simultaneously with AT logs. The RNC logs can be post processed with NEWTUN on a per RNC basis to determine the number of RC (Resource Control) blocks that occurred.

3. Drive the cluster route and stop the RNC logs at the conclusion of drive testing.

10.5.1.3 Blocking OMs Method of Collection: 1. The following OMs measure the airlink resource allocation performance for connection setups and

soft / Softer handoff only. With release 3.0 call admission control is moved from the DO-RNC to the DOM therefore, setup failure OMs due to resource allocation issues should be from DOM specific metrics.

2. In order to setup data collection, follow the guidelines in section 5.4.4 to setup the data collection configuration with the RNCPerfBySectorCarrier_R3.0 template selected.

10.5.2 Blocking KPI Calculation 10.5.2.1 KPI Value and Tolerance Factor This is operator specific and what the System was designed to meet. Need to define different Blocking rates for different entities within the System. Channel Elements, CPU, Connection Limits, Session Limits. Can this be used for troubleshooting as well as provisioning? There are datafill parameters {connection limits} than can affect blocking. FDD for remote sectors also affects this KPI.

10.5.2.2 Calculating KPI from AT Logs 1.From EVDO RFO:

( )

−−−−= lEOFalAccesDOEOF_ParixHandofftoilsDOAccessFaDOAtts

DOBlocksBlocking% Re1

10.5.2.3 Calculating KPI from RNC Logs 1. Process the RNC logs with NEWTUN and open the statistics view.

2. Click on the Access Terminals Data and Session tabs and note the number of blocks total for each.

3. The following equation can be used to calculate the blocking percentage during drive testing.

100% XttemptsTotalCallAcksSessionBloDataBlocksBlocking

+=

10.5.2.4 Calculating KPI from OMs 1. The following formulas can be used to determine the blocking rate at the RNC or the Sector Carrier

level.

2. Connection Setup Failures due to unsuccessful DO-RNC resource allocations can be determined from the OMs as follows:

Percentage of Connection Setup Failures due to Unsuccessful DO-RNC Resources Allocation = 100% * (numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRncResources) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout +


numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted)

3.Connection Setup failures due to unsuccessful DOM resource allocations can be determined from the

following OMs:

Percentage of Connection Setup Failures due to Unsuccessful DOM Resources Allocation = 100% * (numConnectionSetupsBlockedByRn + numConnectionSetupsFailedByRn) / (numConnectionSetupsBlockedByRn + numConnectionSetupsBlockedByRncResources + numConnectionSetupsFailedByRn + numConnectionSetupsFailedByRncResources + numConnSetupsFailedRncTimeout + numConnSetupsFailedRuTimeout + numConnSetupsFailedTccTimeout + numConnSetupsFailedSWError + numConnSetupsAborted)

4. Airlink Resources allocation blocks during connection setup and soft / softer handoff on Primary

DOMs can be determined from the following OMs: Airlink Resource Blocks (Connection Setup + Soft/Softer Handoff) on Primary DOM = numAllocationBlockRnDriverResourceSC + numAllocationBlockRnConnectionLimitSC

5. Airlink Resource allocation blocks during connection setup and soft / softer handoff on Secondary DOMs can be determined from the following OMs: Airlink Resource Blocks (Connection Setup + Soft/Softer Handoff) on Secondary DOM = numAllocationBlockRnDriverResourceSSC + numAllocationBlockRnConnectionLimitSSC

10.5.3 Troubleshooting Guide 1.If the per RNC or Sector Carrier OM’s indicate that blocking exceeds performance thresholds set forth in

the exit criteria, determine the top ten sector carriers for blocking.

2.The top DOM / sector carrier contributors to blocking can be determined from the RNC logs by analyzing the Network Statistics in NEWTUN.


3. To determine the top contributors to blocking from the OMs, evaluate sector performance and determine Sectors which contribute highly to DOM related resource allocation failures. Following Origination Sector (ACH of which used for sending/receiving messages) OMs need to be utilized to isolate high contributors.

sum of OMs:

numATConnectionBlockedByRnSC, numANConnectionBlockedByRnSC and numFCConnectionBlockedByRnSC

need to be evaluated to identify sectors which experience higher blockings. As OMs are pegged against these sectors for connection blocks even-if other sectors may be blocking when additional links were setup during the connection setup, determine the neighboring sectors of these high running sectors.

4. Once neighboring sectors are identified, numAllocationBlockDriverResourceSC OR numAllocationBlockDriverResourceSSC (for lack of Mac Index and CE), numAllocationRnConnectionLimitSC OR numAllocationRnConnectionLimitSSC (for Max. allowed airlinks configuration), numallocationBlockRnSectorCarrierDownSC OR numallocationBlockRnSectorCarrierDownSSC, numAllocationBlockRnModemTimeoutSC OR numAllocationBlockRnModemTimeoutSSC ( for internal timeout between SC and Modem), and numAllocationBlockRnNoConnectionSC OR numAllocationBlockRnNoConnectionSSC (for not being able to find existing connection during softer handoff) OMs need to be evaluated and appropriate actions need to be taken to solve the performance issues.

5. Percentage Contribution of TCC Timeouts to Connection setup failures due to RF Failures (Level 3, per RNC). If this metric is the highest contributor among level 3 metrics, it indicates the issues related to TCC timeouts.

a. TCC timeout happens when Call Control on RNC times out waiting for Traffic Channel Completes message from the AT.

b. Higher TCC timeouts indicates that the RF conditions were not good enough for the TCC message to reach the DOM or TCA message reach the AT or TCC Wait Timer parameter value is not good enough. Link budget need to be verified for the forward and reverse link coverage balance. Power used (configurable parameter) for sending TCA need to be verified

6. Percentage Contribution of Route Update Timeouts to Connection setup failures due to RF Failures (Level 3, per RNC). If this metric is the highest contributor among level 3 metrics, it indicates the issues related to RU timeouts.


a. RU timeout happens when Route Update message is not received by the RNC from AT within a predefined time interval. The timer used for this event is not configurable.

b. Higher RU timeouts indicates that the RF conditions were not good enough for the RU message to reach the DOM. Other contributing factors to this event exists but could not be fixed by us (needs to have design take care of the software). Link budget need to be verified for the forward and reverse link coverage balance

7. Percentage Contribution of Software errors to Connection setup failures due to Miscellaneous Failures (Level 3, per RNC)

a. If this metric is the highest contributor among level 3 metrics, it indicates the issues related to software errors. Software errors occur when the memory pool usage reaches the maximum 1500 connections and failure to create HDR instance occur

8. Percentage Contribution of Software errors to Connection setup failures due to Miscellaneous Failures (Level 3, per RNC)

a. If this metric is the highest contributor among level 3 metrics, it indicates the issues related to aborted connection setups due to following

b. Route update message with Keep = No for all pilots received and Enable Access when last RU has all keep = No is set to false

c. Failure to allocate memory from Heap d. Local close request is generated by operator e. Page timer expires and AT initiated connection setup was not in progress.

9. From CLI determine if the blocking is related to the maximum number of connections being exceeded:

NORTEL-XX>enable NORTEL-XX# show 1xevdo sector carrier <IP> < channel number> <pn offset> counters all

RN Resource Allocation Counters Number of Requests Sent: 2

Successful Allocations: 2 Blocks:

Driver: 0 Connection Limit: 0 Sector Down: 0 Modem Timeout: 0 No Connection: 0 Malformed message: 0

10. The connection limit in the DOM is determined from two parameters: Maximum Forward Distribution

Delay and Connection Limit per sector element. The DOM logic sets the connection limit to the minimum of the two of these parameters. In addition, the RNC Maximum Airlink Connnections per Sector Carrier overrides these two parameters [3].

11. The following is an overview of how the DOM connection limit algorithm works.

12. While setting up a connection, the configured values for Connection Limit per sector at the DOM and Maximum Airlinks per SectorCarrier are checked separately, in the following order:

Step 1: Check against RNC Maximum Airlinks Per SectorCarrier setting. If there is a block, totalBlockedAirlinkRsrcAllocationsSectorCarrier is pegged and setup is aborted.

Step 2: Check against DOM per-sector Connection Limit setting. If there is a block, numAllocationBlockRnConnectionLimitSC is pegged and setup is aborted.


Step 3: Send allocation request to modems (driver). It will check for number of channel elements available. If is a block, numAllocationBlockRnDriverResourceSC is pegged and setup aborted (“setup aborted” meaning the next step is not executed).

Here are some examples to illustrate how this algorithm works:

With RNC Maximum Airlinks per SectorCarrier = 40, the DOM Connection Limit per sector = 48, and Number of Channel Elements = 30, a block at 3rd step of the algorithm will occur for 31st connection

With RNC Maximum Airlinks per SectorCarrier = 30, the DOM Connection Limit per sector = 48, and Number of Channel Elements = 30, a block at 1st step of the algorithm will occur for 31st connection

With RNC Maximum Airlinks per SectorCarrier = 48, the DOM Connection Limit per sector = 30, and Number of Channel Elements = 30, a block at 2nd step of the algorithm will occur for 31st connection

13. Verify from the RNC RF datafill that the Maximum Airlinks per sector carrier is according to the Nortel recommended value of 48.

NORTEL-XX>enable NORTEL-XX#show 1xevdo config

Maximum Airlinks per Sector-Carrier 48 14. If the value is below recommended it can be reset through CLI as follows:

NORTEL-XX(config)#1xevdo max-airlinks-per-sector-carrier 48

15. Verify from the DOM RF datafill that the connection limit is at the recommended value of 48 for the sector carrier being considered. If the value is less than the recommended value use the following CLI command to adjust the value:

NORTEL-XX>enable NORTEL-XX# configure NORTEL-XX(config)# sector-element element1 NORTEL-XX(config-element)# connection limit 48

16. If Blocking persists there may be provisioning issues. Contact network engineering for a verification

provisioning of the network elements.

10.6 Connection Drops


The data collection process to troubleshoot connection drop problem is as follows: • Use one AT and OPTis-M or Invex3G to collect the AT logs. • Setup the log masks to collect all EVDO attributes. If the hybrid mode testing is to be performed

also collect all CDMA log attributes. • Following the process given in the Cluster drive testing section , configure the AT to download a

500 MB test file – wait for a period greater than the data inactivity timer (datafilled in the system to allow the connection to transition to a dormant state between tasks. Repeat the download over and over until at least 30 minutes worth of AT logs have been collected. Please see the example below.


• Drive the cluster where drops are occurring to collect AT logs. Keep the drive test route within one RNC boundary.

10.6.1.2 RNC Logs Method of Collection:

1. Follow the RNC logging setup procedure in section 5.3.1 to configure logging. 2. The most important log to collect is:

Severity

Level Component ID DO-RNC CPUs Message

ID Description

5 Call Control 009 RNSM and SC 423 Connection Drop

3. This message contains useful information such as: drop reason (FTC lost/RTC Lost), frame error rate, and the last 2 sets of TCA and Route Update messages. An example of Message 423 is shown below:

Connection Drop : Drop Reason : No Reverse Traffic Channel Connection Drop : Frame Error Rate 0.9401 (%) Connection Drop : Dump of Last 100 Eb/No Values (0.125 dB) Connection Drop : Last TCA message Details : TCA Seq# 1 sent in response to RouteUpdate Seq# 29 Connection Drop : Details of TCA Seq# 1 Connection Drop : TCA 1 : Channel Included N : Channel 0x0 : Frame Offset 0x8 Connection Drop : TCA 1 : DRCLength 4 : DRC Channel Gain -6 : Ack Channel Gain 6 : NumPilots 1 Connection Drop : TCA 1 : PN Offset 36 : Softer Handoff 0 : MAC Index 63 : DRC Cover 1 : RAB Length 32 : RAB Offset 2 Connection Drop : Details of RouteUpdate Seq# 29 Connection Drop : RU 29 : PN Offset 36 : PN Phase 0 : Strength 2 : Keep Y Connection Drop : RU 29 : Channel Included N : Channel 0|0|73 : RN IP 10.140.96.2 Connection Drop : RU 29 : PN Offset 32 : PN Phase 2048 : Strength 63 : Keep N Connection Drop : RU 29 : Channel Included N : Channel 0|0|73 : RN IP 10.140.96.2 Connection Drop : Previous TCA message Details : TCA Seq# 0 sent in response to RouteUpdate Seq# 27 Connection Drop : Details of TCA Seq# 0 Connection Drop : TCA 0 : Channel Included N : Channel 0x0 : Frame Offset 0x8 Connection Drop : TCA 0 : DRCLength 4 : DRC Channel Gain -6 : Ack Channel Gain 6 : NumPilots 2 Connection Drop : TCA 0 : PN Offset 36 : Softer Handoff 0 : MAC Index 63 : DRC Cover 1 : RAB Length 32 : RAB Offset 2 Connection Drop : TCA 0 : PN Offset 32 : Softer Handoff 1 : MAC Index 62 : DRC Cover 2 : RAB Length 32 : RAB Offset 1 Connection Drop : Details of RouteUpdate Seq# 27 Connection Drop : RU 27 : PN Offset 36 : PN Phase 0 : Strength 6 : Keep Y Connection Drop : RU 27 : Channel Included N : Channel 0|0|73 : RN IP 10.140.96.2 Connection Drop : RU 27 : PN Offset 32 : PN Phase 2048 : Strength 11 : Keep Y Connection Drop : RU 27 : Channel Included N : Channel 0|0|73 : RN IP 10.140.96.2

4. Some other logs that may be useful are:

Severity

Level Component ID DO-RNC

CPUs Message

ID Description

5 Call Control 9 RNSM and SC 213 Connection Closed 9 Call Control 9 RNSM and SC 215 Connection Request 9 Call Control 9 RNSM and SC 216 Traffic Channel Complete Message

16 Call Control 9 RNSM and SC 220 Local Connection Close Event : Reason #

9 Call Control 9 RNSM and SC

221 Rx from AT : Connection Close Message : Reason #



266 Tx to AT : Connection Close Message : Reason Code #

9 Call Control 9 RNSM and SC 269 Traffic Channel Assignment Message 9 Call Control 9 RNSM and SC 326 Add/remove SHO leg


429 Last “N” Eb/No values after a connection drop occurs

10.6.1.3 Dropped Connection OMs Method of Collection: 1.Setup the OM data collection as outlined in section 5.4.4.1 using the RNCPerfByRNC_R3.0

template. 2.To get a statistically significant trend of performance, particularly in a market that is already

commercial it will be useful to collect OMs over an entire week. 3.Dropped connection OMs can be classified by:

• RF Related Drops – No FTC or RTC Lost • Handoff Related Drops – SHO Blocked by RNC Resources, SHO Blocked by RN

(DOM), SHO Failed By RNC Resources, SHO Failed by RN, SHO Failed due to TCC Timeout, or SHO Failed to RNC Timeout

• Miscellaneous Drops – Close Internal Error, Close RLP, State Mismatch, Sector Down, or SSM Disable

4.The Connection Drop OMs are defined below: • numConnectionCloseNoFTC: The number of connections that were closed by DO-RNC

because of indications that there is no active Forward Traffic Channel. • numConnectionCloseRTClost: The number of connections that were closed by DO-

RNC because of indications that the reverse link(s) were lost. • NumRevLinkSHOFailedTccTimeout: The number of reverse link soft handoffs that

failed because the Traffic Channel Complete message was not received from the Access Terminal in time.

• NumConnectionCloseStateMismatch: The number of connections that were closed by DO-RNC due to state mismatch. For example, receiving a Connection Request message from the Access Terminal when the DORNC is in Connected State.

• NumRevLinkSHOfailRncTimeout: The number of reverse link soft handoffs that failed because resource allocation/release on the DO-RNC did not complete in time.

• NumRevLinkSHOFailedByRncResources: The number of reverse link soft handoffs that were failed by DO-RNC resource allocation.

• NumRevLinkSHOFailedByRn: The number of reverse link soft handoffs that were failed by RN resource allocation.

• NumRevLinkSHOBlockedByRn: The number of reverse link soft handoffs that were blocked by RN resource allocation. The resources considered are the channel elements, number of links per sector, and the MACIndices.

• NumRevLinkSHOBlockedByRncResources: The number of reverse link soft handoffs that were blocked by DO-RNC resource allocation. The only resource checked is the number of airlinks per sector.

• NumConnectionCloseFromATError: The number of Connection Close messages from the AT that had a reason code of Error. It is up to the AT implementation.

• NumConnectionCloseSectorDown: The number of connections that were closed by DO-RNC because of indications that a sector associated with the connection has changed state to down.

• NumConnectionCloseRlp: The number of connections that were closed by DO-RNC at the request of the Radio Link Protocol due to errors..

• NumConnectionCloseSlp: The number of connections that were closed by DO-RNC at the request of the Signaling Link Protocol due to errors.


• NumConnectionsOpened: The number of connections opened successfully on this DO-RNC (the AT arrives on the Traffic Channel).

10.6.2 Dropped Connection KPI Calculation 10.6.2.1 KPI Value and Tolerance Factor • The typical Connection Drop Rate value from drive testing is <= 3% ± CI • The typical Connection Drop Rate value from OMs is 4% ± 1%

10.6.2.2 Calculating KPI from AT Logs 1.Normal calls are ended by a Connection Close message from the AT or the AN with the reason =

Normal Close. Connection Closes with Reason other than Normal Close are considered connection drops (e.g. forward Connection Close message with reason = Connection Error or QCP Connection Release packet with reason = System Lost).

2.To calculate the connection drop rate, process the AT logs with EVDO RFO, then look at the Statistics Tool – Calls to find the number of connection drops.

Connection Drop Rate = (Method #1) DO Drops / Number of DO Connections = (Method #1) DO Drops / (DO Drops + DO Completions) or = (Method #2) DO Drops / Effective Calls where: Effective Calls = Total Elapsed Active Time / Average Data Call Holding Time

Example: Method #1 from EVDO RFO Statistics

o Method #1 - Connection Drop Rate = 11 / (11 + 45) = 19.6% Example: Method #2 – Using Data Call Holding time


o Method #2 – Connection Drop Rate o Take the Time value provided in the RLP statistics window and divide it by a data call folding

time (typical length of time a user has an active connection) DCHT = 30s. o 4717s/30s = 157.2 effective calls

o To determine the drop call rate, take the number of drops from the call statistics window (DO Drops) divided by the number of effective calls:

o 11 / 157.2 = 7%

10.6.2.3 Calculating Dropped Connections KPI from RNC Logs 1.After the .bin files are downloaded from the RNC post process the files with NEWTUN. 2.To view the statistics, open the database where the processed results were saved. 3.Open the Access Terminal statistics and sort by the test ATs IMSI or ESN. 4.The Connection Drop Rate is calculated individually for Data and Session related connections and is

given as a Total = Data + Session drops. The following equations show the dropped connection rate from the RNC logs:

100% xTrafficIncompleteAccessesSuccessfulDropsnectionDroppedCon

−=

5. The Dropped Connection Rate should be within the exit criteria specifications

10.6.2.4 Calculating Dropped Connection KPI from OMs Connection drops typically referred to as abnormal connection closes. These typically occur as a result of loosing the RF link or other error conditions

(sNumRFReleatedDrops + cNumRFRelatedDrops) = numConnectionCloseRtcLost + numConnectionCloseNoFtc


(sNumSoftHandoffReleatedDrops + cNumSoftHandoffRelatedDrops) = numRevLinkSHOBlockedByRncResources + numRevLinkSHOBlockedByRn + numRevLinkSHOFailedByRncResources + numRevLinkSHOFailedByRn + numRevLinkSHOFailedTccTimeout + numRevLinkSHOFailRncTimeout (sNumMiscDrops + cNumMiscDrops) = numConnectionCloseInternalError + numConnectionCloseRlp + numConnectionCloseStateMismatch + numConnectionCloseSectorDown + numConnectionCloseSsmDisable Total Dropped Connection Rate (DC) = (cNumRFRelatedDrops + cNumSoftHandoffRelatedDrops + cNumMiscDrops) / (cNumConnectionSetupSuccess + numSToCCrossovers) Where: numSToCCrossovers – number of connections opened before the demarcation point, and were closed normally after the demarcation point.

10.6.3 Dropped Connection Troubleshooting Guide The typical causes of connection drops are as follows: Description Poor RF coverage (probably want to sub-divide into fwd/rvs link, edge of network vs.

internal) Symptoms in Data (AT Logs/RNC Logs/OMs)

AT Logs- Mobile Rx Power < -85 dBm Mobile Tx Power > -10dBm Traffic C/I <= -2dB

RNC Logs- Use EVDO remote optimization scripts to process and analyze RNC logs. The scripts will be available as a module in the Nortel version of EMS parser due for release in June 2005. Currently, the scripts are stand alone and are available on a case by case basis for

Drop


optimization. The following output from the CROS scripts shows a drop due to coverage related issues: RNC :

Drop Drop HMS Fail Drop Estimated File Num Trigger or AF Root Cause

23 49 ####

# Connection Drop Log FCA

Whole call not logged so unclassified

23 50 ####

# Connection Drop Log Drop

Fwd and Rvs coverage both bad, genuine coverage issue Connection Suicide

Poor Forward link coverage typically will result in a dropped connection when the Ec/Io (Strength) of all pilots is less than -9dB. Example from RNC Logs

MOB CSMOpen] : Rx from AT : Route Update Message : Seq# 7 MOB CSMOpen] : Route Update Detail : Processing Seq# 7 : Connected : RefPN 228 : RefStrength 37 : RefKeep Y RefStrength x -0.5 = -18.5dB MOB CSMOpen] : Route Update Detail : RefPN IP Address 10.245.36.39 MOB CSMOpen] : Route Update Detail : Other Pilot# 1 : PNPhase 18942 : Strength 63 : Keep Y : ChanIncluded N : Channel 0|00 Strength x -0.5 = -31.5dB MOB CSMOpen] : Route Update Detail : PNPhase 18942 : PNOffset 296 : RN IP 10.245.37.2 : EPNOin4Chips 0 FlowControl 2 Open] : Rx : Route Update Event event MOB CSMOpen] : Counts after formMobility: Add (0) : Keep (2) : Delete (0) : Keep(2) : Delete (0) SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection ID:0x00103965] : Rx from RN : FTC Stopped Indication Message : Success SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection ID:0x00103965] : Rx from RN : FTC Desired Indication Message : Success SDU CSMOpen] : Setting Active soft handoff leg [IP:10.245.36.39, CR:0x00000aee, RNC Connection ID:0x00103965] : DRC Mask 0x00000004 SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection ID:0x00103965] : Rx from RN : FTC Stopped Indication Message : Success SHO SHOSM_Open] : [IP:10.245.37.2, CR:0x00000aee, RNC Connection ID:0x000f2c54] : Rx from RN : RTC Lost Status Indication Message : DRCMask:0x00000002 SHO SHOSM_Open] : [IP:10.245.36.39, CR:0x00000aee, RNC Connection ID:0x00103965] : Rx from RN : RTC Lost Status Indication Message : DRCMask:0x00000004 CSM CSMOpen] : Rx : Local Connection Close Event : Reason 9 CSM CSMOpen] : Tx to AT : Connection Close Message : Reason Code 2

Solution For areas where dropped connections result from poor RF coverage changes to the underlying 1xRTT network may be required. Any changes to the underlying network will have to be coordinated with the customer.

Comments


Description Slow handoff Symptoms in Data (AT Logs/RNC Logs/OMs)

AT Logs – N/A RNC Logs

Observe the timestamps between last TCA and “Traffic Channel Complete Message from AT not received in time”. If the delta between the two is greater than the TCC wait timer it may be necessary to increase this value especially if the TCC is received after the timeout. OMs Percentage of Reverse Link Soft Handoff due to RF Failure = 100% * numRevLinkSHOFailedTccTimeout / (numRevLinkSHOBlockedByRn + numRevLinkSHOBlockedByRncResources + numRevLinkSHOFailedByRn + numRevLinkSHOFailedByRncResources + numRevLinkSHOFailRncTimeout + numRevLinkSHOFailedTccTimeout + numRevLinkSHOAborted)

Solution Verify Neighborlists, Increase TCC wait Timer. Comments Slow Handoff – When a TCA is sent but no TCC is received. Description Full active set - no mechanism to select best 6 if 7 or more requested ( Fixed in 3.0) Symptoms in Data (AT Logs/RNC Logs/OMs)

AT Logs – RNC Logs – Look for Route update messages in the logs where the number of other pilots listed is six and the SHO SHOSM Failed.


: Route Update Detail : Processing Seq# 142 : Connected : RefPN 476 : RefStrength 7 : RefKeep Y : Route Update Detail : RefPN IP Address 10.245.37.59: Route Update Detail : Other Pilot# 1 : PNPhase 28433 : Strength 42 : Keep Y : ChanIncluded N : Channel 0: Route Update Detail : Other Pilot# 2 : PNPhase 23523 : Strength 38 : Keep Y : ChanIncluded N : Channel 0: Route Update Detail : Other Pilot# 3 : PNPhase 20468 : Strength 47 : Keep Y : ChanIncluded N : Channel 0: Route Update Detail : Other Pilot# 4 : PNPhase 26139 : Strength 63 : Keep Y : ChanIncluded N : Channel 0: Route Update Detail : Other Pilot# 5 : PNPhase 27183 : Strength 63 : Keep Y : ChanIncluded N : Channel 0: Route Update Detail : Other Pilot# 6 : PNPhase 27904 : Strength 63 : Keep Y : ChanIncluded N : Channel 0

: Route Update Detail : PNOffset 444 : PNPhase 28433 : RN IP 10.245.37.61 : EPNOin4Chips 7 : Route Update Detail : PNOffset 368 : PNPhase 23523 : RN IP 10.245.37.72 : EPNOin4Chips 0 : Route Update Detail : PNOffset 320 : PNPhase 20468 : RN IP 10.245.37.30 : EPNOin4Chips 0 : Route Update Detail : PNOffset 408 : PNPhase 26139 : RN IP 10.245.37.67 : EPNOin4Chips 9 : Route Update Detail : PNOffset 424 : PNPhase 27183 : RN IP 10.245.37.71 : EPNOin4Chips 14 : Route Update Detail : PNOffset 436 : PNPhase 27904 : RN IP 10.245.37.61 : EPNOin4Chips 3 : Rx : Route Update Event event : Created PBTSCM node [IP:10.245.37.72, CR:0x00000aee : Created PBTSCM node [IP:10.245.37.61, CR:0x00000aee : Created PBTSCM node [IP:10.245.37.30, CR:0x00000aee : Created PBTSCM node [IP:10.245.37.67, CR:0x00000aee : Created PBTSCM node [IP:10.245.37.71, CR:0x00000aee: Counts after formMobility: Add (6) : Keep (1) : Delete (0)

OMs – numRevLinkSHOFailedByRn This problem is not readily recognized in the OMs as it will be aggregated with other SHO failure reasons.

Solution Patch 2.1.0.25 (Performance Patch) will fix the 7th Pilot issue by limiting the number of pilots to be added to the active set such that it never exceeds 6. At most (6 – (Current Active Set Size)) pilots may be added to the Active Set. Pilots will be chosen according to signal strength.

Comments Description InterRNCActiveHandoff Optimization Unhomed or Unknown Pilot Symptoms in Data (AT Logs/RNC Logs/OMs)

Unknown Pilot - If the received RouteUpdate message contains a pilot that does not appear in the neighbor-list of any of the pilots in the AT’s current active-set or in the neighbor-list of any of the pilots in the RouteUpdate message being processed, the pilot will be ignored. The soft handoff, however, will not fail just because of the presence of the unknown pilot in the RouteUpdate message. Unhomed Pilot - If the received RouteUpdate message contains a pilot that is in the neighborlist of any of the pilots in the AT’s current active-set or in the neighbor-list of any of the pilots in the RouteUpdate message being processed but the DOM “owning the sector” is not homed to the DO-RNC, the pilot will be ignored. The soft handoff, however, will not fail just because of the presence of the unhomed pilot in the RouteUpdate message. RNC Logs

In the logs above the Route update is for PNs 93, 45, and 426. The source has no knowledge of the PN’s 93 and 426. It is likely that these PNs live on another RNC.


OM’s Number of Times an Unknown Pilot in RU when a Primary Pilot Is Strongest = numPilotLookupFailuresUnknownPilotSC Number of Times an Unknown Pilot in RU when a Primary Pilot Is Strongest = numPilotLookupFailuresUnknownPilotSSC When this OM is pegged, the neighbor list needs to be examined and optimized. The pilot lookup failure logs from the DO-RNC should then be used for detailed information. Number of Times an Unhomed Pilot in RU when a Primary Pilot Is Strongest = numPilotLookupFailuresRNNotHomedSC Number of Times an Unhomed Pilot in RU when a Primary Pilot Is Strongest = numPilotLookupFailuresRNNotHomedSSC When this OM is pegged, the secondary homing configuration needs to be examined and optimized. The pilot lookup failure logs from the DO-RNC should then be used for detailed information. • numAllocationAttemptsTxRnSC and numRevLinkSHOAddAttemptsSC • numPilotLookupFailuresRNNotHomedSC • numAllocationAttemptsTxRnSSC and numRevLinkSHOAddAttemptsSSC • numPilotLookupFailuresRNNotHomedSSC

Solution Add source RNC as secondary RNC in interfering DOM’s secondary Homing table. Add adjacent (interfering) DOM to the neighborlist of source sector carrier.

Comments In 3.0, the existence of alien pilots in the route update message will not cause the call to fail but will contribute to interference which can cause drops and impede maximum sustained data rates.

Description

Hybrid AT non-returning tune away

Symptoms in Data (AT Logs/RNC Logs/OMs)

AT Logs – In a long tune away the AT will release the EVDO connection after being tuned away for longer than 5 seconds. To recognize long tuneaways from the message view in EVDO set the message filter for the following: 1. CDMA 2000 f-csch (Paging Messages) - All 2. CDMA 2000 forward Traffic Channel – Neighborlist Update 3. IS856 Access – (Connection Request, Route Update) 4. IS856 Control Channel Directed – (Traffic Channel Assignment, ACAck, 5. IS856 Control Channel Broadcast – All overhead messages 6. IS856 Forward Traffic – (Connection Close, Traffic Channel Assignment, 7. IS856 Reverse Traffic – (Connection Close, Route Update, Traffic Channel Complete 8. QCP EVDO – Connection Release, ARQ Effective Receive Rate, Connection Attempt

From RFO Map View:


Example from Grid View:

Example from Message View below:

AT has lost supervisory communication with the EVDO network

No EVDO RF statistics for 5 seconds – tuned away


RNC Logs- In the RNC logs long tuneaways can typically be identified by an “FTC Stopped Indication Message (Null DRC from AT)” followed by at least 5.12 seconds of inactivity in the RNC messaging. In the example below it can seen that the stopped indication is sent and roughly 5 seconds later the AT communication is re-established with the DO network but the RTC is lost as a result. Example from RNC logs:

16 03-10-05 03:45:10.972

SHO SHOSM_Open] : [IP:10.245.37.59, CR:0x00000aee, RNC Connection ID:0x000d3650] : Rx from RN : FTC Stopped Indication Message : Success

51 03-10-05 03:45:11.002

CSM CSMOpen] : Periodic Counters : RLP Tx 105.0 : SLP Tx 21.0 : RLP Rx 154.0 : SLP Rx 32.0 : RLP to A10 152.0

ca 03-10-05 03:45:11.002

CSM CSMOpen] : Periodic Counters : Connection Duration : 00:00:22.579

(No activity) (Tuned Away to 1x)

57 03-10-05 03:45:15.882

SHO SHOSM_Open] : [IP:10.245.37.59, CR:0x00000aee, RNC Connection ID:0x000d3650] : Rx from RN : RTC Lost Status Indication Message : DRCMask:0x00000002

50 03-10-05 03:45:15.882

CSM CSMOpen] : Rx : Local Connection Close Event : Reason 9 (Reason Code 9 = RTC Lost)

08 03-10-05 03:45:15.882

CSM CSMOpen] : Tx to AT : Connection Close Message : Reason Code 2

OMs – numConnectionClosenoFTC – Pegged when an FTC Stopped is received and an FTC Desired message is not received within 5 seconds

Solution Currently there is no complete fix for a drop due to a long tune away. Some theories on insuring that the 1x Channel lists are consistent from BTS to BTS.

Comments Description SHO block Symptoms in Data (AT Logs/RNC Logs/OMs)

RNC Logs- In the RNC logs a drop due to a SHO block may be identified by a RC lookup fail. Example from RNC Logs:

MOB CSMOpen] : Rx from AT : Route Update Message : Seq# 68 MOB CSMOpen] : Route Update Detail : Processing Seq# 68 : Connected : RefPN 72 : RefStrength 27 MOB CSMOpen] : Route Update Detail : RefPN IP Address 10.245.37.41 MOB CSMOpen] : Route Update Detail : Other Pilot# 1 : PNPhase 2299 : Strength 19 : Keep Y : ChanI MOB CSMOpen] : RC lookup failed : Dropping pilot from Route Update (PNOffset: 36, Phase: 2299

OMs- numRevLinkSHOFailedByRN – number of reverse link soft handoffs failed due to resource allocation. numRevLinkSHOBloakedByRN - The number of reverse link softhandoffs blocked by RN resource allocation. numRevLinkSHOBlockedbyRNCResources – The number of reverse link softhandoffs that were blocked by DO-RNC resource allocation.

Solution Comments


Description Hand down algorithm to 1xRTT kicks in and AT doesn't manage to send connection close

Symptoms in Data (AT Logs/RNC Logs/OMs)

Solution Comments Description Loss of Reverse Traffic Channel Symptoms in Data (AT Logs/RNC Logs/OMs)

A reverse traffic channel is determined lost when the RL Fade timer expires on the DOM (there is one RL Fade timer for each of the soft/softer handoff leg). The RL Fade timer is triggered when the DOM is unable to successfully demodulate non-erased DRC decisions from the AT. This timer is reset when non-erased DRCs are received from the AT (the DRC cover for the DOM is not matched for this indication). When RTCLost indication has been received from all the Down Legs associated with the connection, it is treated as a loss of the reverse link to the AN and the connection is closed. AT Logs Loss of the Reverse Traffic Channel is hard to be sure of when looking at the AT logs. The best way to determine that this has occurred is to examine the Power Log message and see that the AT’s transceiver has turned off. RNC Logs

OMs numConnectionCloseRtcLost numConnectionCloseRtcLostSlot

Solution Increase the Reverse Link Fade Timer. Care must be taken when doing this as it may waste resources when adequate RF does not exist to overcome a fading.

Comments


Description Missing neighbor Symptoms in Data (AT Logs/RNC Logs/OMs)

AT Logs- N/A RNC Logs –

In the logs it can be seen from the Route Update Detail that the IP lookup failed for PN 348 and PN 351.

Solution If the drop is in fact due to a missing neighbor the neighborlists for the reference site and its neighboring sectors should be evaluated to insure all the required PN’s are included and that the proper IP’s are also included. Alternatively, if the requested PN’s are from sites in an adjacent RNC insure that those DOMs have the source RNC datafilled as a secondary RNC and that this DOM is in the neighborlist of the source DOM.

Comments


Description PDSN Release Symptoms in Data (AT Logs/RNC Logs/OMs)

PDSN release A10 de registration AT Logs RNC Logs

When a connection to the AT is opened LCP requests are sent over the open channel, but are not responded to since there is no PPP end point at the AT. The connection will stay up until the PDSN times out (typically about 30 seconds +). This condition will lead to an increase in dropped calls. OMs Percentage of A10 Connection Close due to PDSN Initiated Close = 100% * numA10ClosedPDSNInitiatedRelease / totalA10Closed

Solution Insure that the UA10 minimization feature is enabled. The UA10 feature will prevent unnecessary setup of A10 and thus PDSN timeouts. With UA10 enabled the AT will only establish a PPP session with the PDSN when the following events occur: • ULN – unsolicited Location Notification after session parameters are

negotiated • Dormant RNC handoff when IP of PDSN that has the PPP session is passed

in the A13 Response message • In Mobile IP terminal when PPP re synch (Connection Request on dormant

or inter technology handoffs. Comments Characterized by A10 de registration after 30+ seconds.


Description GBU (Montana drop-off) Symptoms in Data (AT Logs/RNC Logs/OMs)

The delta in PN Ec/Io from best to worst is >=14dB. RNC Logs

Max PN Ec/Io = -7.5dB Worst PN Ec/Io = -31.5dB delta = 24dB

Solution No Current Solution Comments GBU (the good, the bad, and the ugly) – usually results when the power control bits are not

read accurately.

10.7 FL Sector Switching Success Rate


Not currently available from AT logs

10.7.1.2 RNC Logs Not Currently Available from Statistics in NEWTUN

10.7.1.3 OMs Method of Collection

1.Setup the OM data collection as outlined in section 5.4.4.1 using the RNCPerfByRNC_R3.0 template. 2.To get a statistically significant trend of performance, particularly in a market that is already

commercial it will be useful to collect OMs over an entire week. 3.If the market is not commercial perform drive test of each cluster on the golden route.

10.7.2 Forward Sector Switching Success Rate KPI Calculation

10.7.2.1 KPI Value and Tolerance Factor 1.Forward Sector Switching Success Rate should be in >=95% for the 90th percentile of all attempts.

10.7.2.2 Calculating KPI from AT Logs 10.7.2.3 Calculating KPI from RNC Logs 10.7.2.4 Calculating KPI from OMs 1. The following metrics only measure the forward sector switches between BTSs. Forward Sector

Switching between adjacent sectors is not visible to the DO-RNC.

Fwd Serving Sector Switching Successes = numTotalSuccessSHO


Fwd Serving Sector Switching Failures = numConnectionCloseNoFtc Therefore, Fwd Serving Sector Switching Success Rate = numTotalSuccessSHO / (numTotalSuccessSHO + numConnectionCloseNoFtc)


Description FTC Desired Not Received in Time Symptoms in Data (AT Logs/RNC Logs/OMs)

OMs numConnectionCloseNoFTC When the currently serving DOM either receives a null rate DRC, null DRC cover, DRC cover from another DOM or fails to demodulate a non-erased DRC from the AT (the QCOM ASIC driver on the DOM uses a filter to avoid being excessively reactive to these situations while at the same time ensuring that a legitimate DRC switch indication is handled in time), the Forward Traffic Channel is stopped and an FTCStopped indication is sent to RNC. On receiving the FTCStopped indication, the RNC triggers the FTCDesiredWaitTimer (this would apply in the case of the connection drop – in normal DRC switching scenarios, the FtcStopped indication could be paired with a pending FtcDesired indication from another leg). If no FTCDesired indication (triggered by DOM receiving a valid non-null or nonerased DRC from the AT, either on the same leg or a different leg) is received by the time the FTCDesiredWaitTimer expires, the RNC closes the connection and pegs this OM. The same scenarios that cause this OM to peg could also cause (numConnectionCloseRtcLost and numConnectionCloseRtcLostSlot) to peg, however only one reason will be pegged for a single connection close depending on whichever timer expires first. If an open connection needs to be closed, the attributes (numConnectionCloseToAtError and numConnectionCloseToAtErrorSlot) will also be incremented. AT Logs RNC Logs Look for FTC Desired Not Received in Time.

Solution Increase the FTC Desired Wait Timer. Insure neighborlist integrity and reciprocity between BTS.

Comments

10.8 FL Sector Switching Time

10.8.1 Data Collection Methods 10.8.1.1 AT Logs Method of Collection 1. Set AT to download a large file (>= 100 MB) over and over. 2. Drive test and collect AT log. 3. Collect all EVDO logging attributes

10.8.1.2 RNC Logs Method of Collection


N/A

10.8.1.3 OMs Forward Sector Switching performance metrics are only available for DOM to DOM (inter-DOM) sector switching. Intra DOM sector switching is transparent to the DO-RNC. The following OMs are available to calculate the switching success or failures between BTSs.

CLI can be used to determine the DRC switching successes between DOMs on a network or Session specific basis as follows:

NORTEL-XX#show 1xevdo counters all (Network wide) NORTEL-XX#show 1xevdo session <UATI> counters all (UATI / Session specific)


From [12]: <200 ms for 90th percentile?

10.8.2.2 Calculating KPI from AT Logs Currently, the forward link sector switching time can not be determined accurately from the AT logs. It has to be done using RNC logs.

10.8.2.3 Calculating KPI from RNC Logs

From [12]: 1. In order to determine the amount of time required for the AT to switch between DOMs in a soft switchover event, it is necessary to determine when one DOM stops sending data and the new DOM begins. 2. The key indicators of a soft forward sector switch are as follows:

a. FTC Stopped or FTC Desired – It should be noted that the initiation of a sector switching event can be demarcated by either of the above messages appearing first, although FTC Stopped and FTC Desired messages should have different RN IP addresses to denote two distinct DOMs. b. The completion of the forward sector switching event concludes with the Setting Active soft handoff leg [IP (same as that in the corresponding FTC Desired Message)]

3. Using a Unix emulator such as Cygwin perform: a. Egrep “(FTC Stopped |FTC Desired \Setting Active soft handoff leg)” This will parse out only the commands in the RNC logs where commands are sent by the AT telling the DOM to stop sending data and the address of the new DOM. b. Soft events are denoted by the UATI of the AT followed in the same line by the IP address of the DOM sending the FTC Stopped or Desired message. Soft events are identified by seeing different IP addresses for the FTC Stopped and FTC Desired DOMs.

4. Calculate the time delta between (FTC Stopped or the FTC Desired message, whichever comes first) and the Setting Active soft handoff leg message for the UATI of the AT under test for all soft switchover events and calculate the 90% percentile value

Message Router Consumer Rx from RN IP Address 47.135.201.245 FTC Desired Indication Message [IP 47.135.201.245, CR 0xc65, RNC Connection ID 0x300ca] Rx from R FTC Desired Indication Message [UATI 0xfc03e9] Received Stopped Indication Message Router Consumer Received TLV 18 from RN IP Address 47.135.201.249 Destinatio127.1.3.1 on SF Message Router Consumer Rx from RN IP Address 47.135.201.249 FTC Stopped Indication Message [IP 47.135.201.249, CR 0xc65, RNC Connection ID 0x300c8] Rx from R FTC Stopped Indication Message Setting Active soft handoff leg [IP 47.135.201.245, CR 0xc65, RNC Connection ID 0x300ca] DRC Mask 0x4 [IP 47.135.201.249, CR 0xc65, RNC Connection ID 0x300c8] Tx to RN Flush FTC Queue Request Message [UATI 0xfc03e9] Received a Set Desired indication, qId 2, Frames 500 [UATI 0xfc03e9] Received a Start Transmission indication RNC Log Example for Forward Switching


10.8.2.4 Calculating KPI from OMs Forward Sector Switching performance metrics are only available for DOM to DOM (inter-DOM) sector switching. Intra DOM sector switching is transparent to the DO-RNC. The following OMs are available to calculate the switching success or failures between BTSs.

Fwd Serving Sector Switching Successes = numTotalSuccessSHO Fwd Serving Sector Switching Failures = numConnectionCloseNoFtc

10.8.3 Troubleshooting Guide 1. Cases where forward sector switches fail are most likely due to an RC resource allocation failure, TCC

(from AT) not received in time, or a tuneaway.

2. If the RNC logs show a high rate of TCC not received in time messages during forward sector switchovers trying increasing the TCC wait timer.

TCC Wait Timer

Range Unit Recommended

1000 – 30000 1/1000 seconds

5000

NORTEL-07>enable

NORTEL-07#configure

Then the TCC Wait Timer can be changed through the following command

NORTEL-07(config)#1xevdo timer tcc-wait <TCC Wait Timer setting>

To verify the timer was changed, go back to the “enable” state and enter the following,

NORTEL-07#show 1xevdo config

The parameter is listed as “CSM Traffic Channel Complete Wait Timer”.

10.9 RL SHO Success Rate In EVDO, reverse link soft handoff is initiated by the AT by sending a Route Update message requesting the addition or removal of pilots from the Active set. Each route update is counted as a handoff attempt regardless of the number of pilots in the message.


Method of Collection: 1.Configure the AT log data collection tools to perform an FTP download of a large file (100Mbytes). 2.Perform a drive of the Cluster golden route 3.Post Process the collected data with EVDO RFO. 4.There are currently not any statistics in EVDO RFO for determining the RL SHO Success rate but

the number of attempts and successes are defined as follows: • Attempts = number of Route Updates on Reverse Traffic Channel with “Num Pilots” > 0 • Successes = number of Traffic Channel Assignments on Forward Traffic channel (with pilots

from previous Route update) followed by the corresponding Traffic Channel Complete message sent on the Reverse Traffic Channel.

10.9.1.2 RNC Logs There is currently no easy way to determine the RL SHO success rate from RNC logs. However RNC logs should be collected such that timing issues or resource allocation issues between the AT and the RNC can be diagnosed.


Method of Collection: 1. Follow RNC logging setup given in section 5.3.1 of this document. 2. Collect logs at Severity level : 8

10.9.1.3 OMs Method of Collection:

1. Follow the procedure in section 5.4.4.1 for configuring OM collection 2. The following OMs are available to measure the RL SHO attempts, successes and Success Rate:

Rev Link SHO Attempts = numRevLinkSHOAttempts Rev Link SHO Successes = numRevLinkSHOSuccess Successful Reverse Link SHO Rate= numRevLinkSHOSuccess / numRevLinkSHOAttempts

3. Reverse Link Soft handoff OMs can be determined from the CLI prompt using the following commands: NORTEL-XX# enable NORTEL-XX# show 1xevdo counters all NORTEL-XX# show 1xevdo session <UATI> counter all (per UATI based for specific ATs)

10.9.2 KPI Calculation 10.9.2.1 KPI Value and Tolerance Factor Shall be >= 99% as measured via OMs.

10.9.2.2 Calculating KPI from AT Logs The following procedure can be used to estimate the RL SHO Success Rate from the AT logs: 1. Open the Message View in EVDO RFO followed by the Message Set Editor. 2. From the Message Set editor select the following IS856 messages:

a. Access – i. Connection Request

b. Forward Traffic – i. Traffic Channel Assignment

ii. Connection Close iii. RTC Ack

c. Reverse Traffic – i. Route Update

ii. Traffic Channel Complete iii. Connection Close

d. QCP EVDO i. Connection Release

3. Export the Message Set to Excel 4. Use Auto Filter to filter on Reverse Traffic Channel 5. Under “MessageName” use the “Countif” worksheet function to determine the number of Route

Updates and the number of Traffic Channel Completes and the number of Connection Requests. 6. There is typically a route update sent after the mobile is on traffic that is not necessarily for SHO. In

order to make the Rate estimate as accurate as possible count the number of CR to account for it and subtract it from the total RUs.

( )

( ) ( )[ ]

−= CRqConnectionRUAttsTCCSuccessesSuccessSHORL #Re#

#%__

10.9.2.3 Calculating KPI from RNC Logs N/A

10.9.2.4 Calculating KPI from OMs Successful Reverse Link SHO Rate= numRevLinkSHOSuccess / numRevLinkSHOAttempts

The RL SHO Success rate can be determined from CLI using:


NORTEL-XX# show 1xevdo counters all (Network specific)

NORTEL-XX# show 1xevdo session <UATI> counters all (AT / Session specific statistics)

RL Soft Handoff Success Rate = Successes / Attempts

10.9.3 Troubleshooting Guide 1. If the RL SHO success rate is less than 99% according to the OMs, the following events may have

caused the failure: Resource Blocks:

RL Soft Handoffs Blocked by RN (Links per sector, MAC indices, or channel elements)

RL Soft Handoffs Blocked by RNC Resources (Number of Airlinks per sector)

Allocation Failures:

RL Soft Handoffs Failed by RN (DOM Resource allocation failure)

RL Soft Handoffs Failed by RNC Resources (RNC Resource allocation failure)

RL Soft Handoffs Fail RNC Timeout (TCC was not received from the AT in time)

2. From the OMs there are failure measurements that will help pin point failure reasons

There is no individual OM identifying the blocking or failure reasons such as no MAC Index, no reverse link demodulator, or no forward link queue. Percentage of Reverse Link Soft Handoff due to Unsuccessful DOM Resources Allocation = 100% * (numRevLinkSHOBlockedByRn + numRevLinkSHOFailedByRn) / (numRevLinkSHOBlockedByRn + numRevLinkSHOBlockedByRncResources + numRevLinkSHOFailedByRn + numRevLinkSHOFailedByRncResources + numRevLinkSHOFailRncTimeout + numRevLinkSHOFailedTccTimeout + numRevLinkSHOAborted) Percentage of Reverse Link Soft Handoff due to RF Failure = 100% * numRevLinkSHOFailedTccTimeout / (numRevLinkSHOBlockedByRn + numRevLinkSHOBlockedByRncResources + numRevLinkSHOFailedByRn + numRevLinkSHOFailedByRncResources + numRevLinkSHOFailRncTimeout + numRevLinkSHOFailedTccTimeout + numRevLinkSHOAborted)

3. From the CLI prompt perform the following commands to determine the root cause of handoff failures:

NORTEL-XX# show 1xevdo counters all

Reverse Link Soft Handoffs :- Attempts : 0 Successes : 0 Blocked by RN : 0 Blocked by RNC : 0 Failed by RN : 0


Failed by RNC : 0 Failed because of no TCC : 0 Failed because of RNC Resource Timeout : 0

4. If any of the RL SHO failures are due to blocking refer to the troubleshooting section on call blocking.

5. The unsuccessful DOM resources allocation could be the result of improper setting of the maximum airlinks per sector attribute in the DOM. See Per Sector Airlink Resources Allocation on page 13-1 and DOM Resources Usage on page 16-1 of reference [18].

6. Any RL SHO failures due to Failed RNC Timeouts can only be due to the DOM being on a software load prior to Release 3.0.

7. TCC timeouts may also be enhanced by increasing the TCC Wait Timer using CLI. The timer can be configured in increments of 1 millisecond between 1000 to 30000 milliseconds. (default 5000)

NORTEL-XX> enable

NORTEL-XX#(configure)

NORTEL-XX(config) #1xevdo timer tcc-wait <time>

10.10 FL Per-User Throughput

10.10.1 Data Collection Methods 10.10.1.1 AT Logs Per user FL throughput is determined by the RF characteristics reported by the AT. Prior to beginning any throughput tests of the EVDO network, any pilot pollution and coverage issues associated with the underlying CDMA network should be addressed.

Forward Link per user throughput data can be captured for the physical layer as well as the application layer using Optis-M or Invex3G and post processing the log files with RFO. Tests of forward link per user throughput can be performed in either mobility or stationary conditions. Stationary testing is typically done to determine the maximum achievable rate per user.

Method of Collection 1. Set up the data collection tool (Optis-M or Invex3G) to perform ftp downloads of a 5MB

(5000000Bytes) test file.

2. Refer to [11], [12], and chapter 5.3 of this document for specific details regarding setting up the test equipment to test throughput.

10.10.1.2 TCP Trace Logs 1.Ethereal or WinDump should be used to capture traces of TCP layer performance data and protocols

being negotiated during PPP session establishment.

2.Capture of TCP trace logs at PPP session setup will allow engineers to determine if IP header compression is enabled on the PDSN.

Method of Collection 3. Download and install Ethereal (v 0.10.10) http://www.ethereal.com/distribution/win32/ on the collection computer.

4. Start a new live capture prior to connecting the AT to the network.


5. Connect the AT to the network and perform an FTP download.

6. At the end of the download stop the TCP trace capture. Ethereal will process the trace logs and display the protocol messages in the workspace.

7. Insure that within the PPP IPCP protocol configuration request message options field, that IP Header compression is not a requested option or that it is being rejected. (see below)

Example: No Request for Header Compression Option


Example: Request for Header Compression Option Rejected by PDSN

10.10.1.3 RNC Logs Data available in the RNC logs does not contain any useful information about user throughput.

10.10.1.4 OMs There are no OMs currently available for measuring per user throughput

10.10.2 KPI Calculation

10.10.2.1 KPI Value and Tolerance Factor The KPI for FL per user throughput is measured at the application layer in order to determine the end user experience. However, requested data rate is based on the DRC request rate and is determined by the AT pursuant to local RF conditions. In general, data rates of 1.5Mbps or greater should be achievable where the average DRC rate is between 2.457Mbps and 1.2288kbps. The table shown below shows estimated values of equivalent application layer throughput based on a percentage of DRC rate requests and other factors.

% Rate Requested DRC Error Rate DRC Rate (Mbps) Application Throughput (Mbps)

Delta from Requested Rate

100 10% 2.457 1.83 23.75% 80 10% 2.457 1.464 20 10% 1.2288 0.183 25.5%

80 10% 2.457 1.464 10 10% 1.843 0.1372 10 10% 1.2288 0.0915

21.7%

10.10.2.2 Calculating KPI from AT Logs 1. After AT log data collection is complete, use EVDO RFO to post process the log files.

2. For application layer throughput statistics, in RFO select, ”Statistics tool” followed by “Data Statistics” then “Application”. (see below)

3. For physical layer throughput statistics and slot utilization rate, in RFO select the “Statistics Tool” followed by “Data Statistics” and “Forward Rate”. (see below)


4. The Graph view in RFO can also be used to show Application, and Physical layer throughput versus time.

Example: Application Throughput Statistics

Formula: Application throughput (kbps) = (RX Data (Bytes) x 8) / 1000) / RX Time (sec)

Example: Physical Layer Throughput Statistics

Average Physical Layer Throughput => Avg Actual Data Rate (kbps)

10.10.2.3 Calculating KPI from RNC Logs Not Applicable – RNC logs contain no useable information for per user throughput at this time.

10.10.2.4 Calculating KPI from OMs There are no specific OMs available for calculating per user throughput however, it can be estimated using the FTC scheduler OM “perfFLthruput”. This OM is averaged over a sliding 10 second window. The reported throughput is the average over the last 10 seconds of the data call.

To estimate the per user throughput perform the following:

1. From the data collected remove data from the sectors that only have a few active airlinks (IS856SecElActiveConnections <=2 This OM is not currently collected but can be enabled by adding a new data collection template) It is necessary to remove this information because the estimated throughput will be artificially low because of the ratio of data to be sent to total number of slots.

2. Use the remaining data to estimate the FL user throughput:

Formula: Average forward link per user Physical Layer throughput = Average of [Forward link sector Physical Layer throughput (perfFLThruput) / Number of Airlinks on the sector] calculated from the remaining data from the previous step.

3. The value obtained using this method is an estimate over all users. The individual estimate could deviate from the estimate significantly.

10.10.3 Troubleshooting Guide 1. Follow the testing and troubleshooting flow chart given in Appendix A.


2. It should be expected that user throughput will suffer in areas at the fringe of coverage, inter RNC borders, or where user traffic is high (scheduling contention).

3. Verify that he DRC requested rate and the Application Rate track within about 15% of one another. In addition insure that the C/I and PER are valid for the data rates being requested. Also verify from the “EVDO Rate Statistics” in XCAL, that the majority of DRC requests are between 1.2288 and 2.457Mbps.

Example: Avg. Effective DRC, Application RX Bytes, C/I1, and TC_PER


Example: Grid View App. RX Rate, Avg. Effective DRC, C/I, and TC_PER

Formula % difference DRC and App rate: ((Avg. Effective DRC – App Rx Rate)/Avg. Effective DRC)

4. If the C/I and PER values are ideal (i.e. between 8 – 12dB) and the DRC rate is low, verify the port settings in the data collection tool are correct. If using XCAL-DO go to My Computer Properties Hardware Device Manager Modems and Ports. Verify the COM ports for the modem are those configured in XCAL COM Port and Data Port Settings screens. For Optis-M, port configuration is automatically detected.


Example: Device Manager Modem Port Configurations

Example: Port Configuration in XCAL-DO

5. If DRC and C/I are within ideal limits, verify Application throughput is between 1.5 and 2.1Mbps.

Data Port

Control Port / Diagnostics Port


Example: Application Throughput from XCAL-DO

6. If the application throughput seen with the data collection tool is lower than expected. Perform a DOS FTP download to confirm the achievable throughput on the FL. If the DOS throughput agrees with the tool, verify that IP header compression is disabled. Go to: Network and Dialup Connections “PPP Adapter” Advanced TCP/IP properties.

Example: IP Header Compression

7. To determine if there are per user throughput problems where coverage is generally expected to be good, post process collected AT logs with RFO and verify that the C2I and DRC request rates track and that the PER does not exceed 1%. Remember that the mobile will decrease the DRC request rate to maintain PER at 1%. (see below)

8. Verify that the application throughput is approximately 15% less than that of the Forward physical rate. An inspection of the graph view in RFO should indicate that application throughput trends less than that of the physical layer. In addition to the graph view, an estimate of the difference in throughput can be determined from the grid view by filtering out all the blank values in the Application RX Rate column (see below).


Example: Graph View App RX Rate vs. Fwd Physical Data Rate

9. If the physical layer throughput is less than that of the application layer throughput and slot utilization is low, software compression is probably enabled in the network and dial up settings on the collection tool / laptop. Software compression will also exaggerate the actual application layer throughput. Software compression should be disabled prior to performing further tests.


Example: Grid View of Application layer vs. Physical Layer Statistics

Example Calculation: ((Fwd Physical Data RX Rate – Application RX Rate) / Forward Physical Data Rate) = 13.5%

10. An initial review of the application statistics in RFO may NOT indicate an immediate problem with achievable data rate. However, a review of the physical layer statistics may show disparity in throughput between the physical and application layer. Application layer throughput should always be less than the Physical Layer rate.

Example: Data Statistics RX Active Throughput (App. Data Rate)

Expected


Example: Data Statistics Fwd Physical Data RX Rate

11. Slot utilization indicates if the application is making efficient use of the physical resources (time slots). In ideal RF conditions slot utilization should be between 65 – 90%. (See below)

Example: Graph View of Application, Physical layer Throughput, and Avg. Effective DRC

Example: Graph View of Poor Fwd Slot Utilization %

Application RX Rate is higher than the Physical layer throughput rate. Compression could be enabled.


Example: Graph View of Good Fwd Slot Utilization %

App RX Rate (kbps) = Average Application Layer throughput per reporting period

Forward Physical Data RX Rate (kbps) = Instantaneous Physical Layer throughput from Forward Rate Statistics V2 message. Indicative of how many packets are sent at a given rate every second. Packets per rate are accumulative on a per second basis.

Average Effective DRC Rate = Data rate requested by the AT based on measured C2I (RF conditions).

Fwd Slot Utilization % = Slot utilization is low due to inefficient use of available slots. In ideal RF conditions where 1 user has the full queue of the scheduler, slot utilization should be in the 80 -90% range.

Example: Fwd Rate Statistics V2 Message

Poor Slot Utilization ~ 2% Better Slot Utilization ~37%


12. Low slot utilization with relatively good application layer throughput may be indicative of software compression. Otherwise poor slot utilization is likely due to poor RF conditions (i.e. PER, C/I, TX poor, etc), AT problem, or many users accessing the sector.

13. If RF problems (i.e. Dropped connections and Access Failures) are excessive, refer to: Section 7.4 and 7.6 for tips on how to recognize and address these problems.

14. It is important to remember that the proportional fairness scheduler will impact throughput as more users access the network (scheduling contention). The scheduler divides time on the system amongst users according to current RF conditions, data rate requests, and past data rate history. The following equation describes how the scheduler determines the throughput per slot per user. .

Where:

TK(n) = mobile #k’s throughput (T) history for time slot n

TC is a time constant

RK(n) = the rate at which the user is being served in slot n

15. To determine the number of users / links in use at a particular time in the AT log file, open the “Message View” in RFO and look for the Quick Config message. This message gives an indication of how many links are attached to a sector while data throughput testing is being performed.

16. The RPC count indicates the number of users attached to the sector at the time. According to the example below, there are four links in use.

Example: Quick Config Message

( ) ,...2,1,0 ),(111)1( =∗+∗

−=+ nnR

tnT

tnT k

ck

ck


17. Given that the maximum forward data rate is contingent upon the reverse link data rate capability, a verification of Forward Physical Rate to reverse data rate ratio should be made. A ratio of no more than 63:1 (optimally 40:1)is sufficient to support data rate requests in excess of 2.0Mbps.

Example: Fwd Physical Rate to reverse data rate ratio

Formula: Ratio = Fwd Physical Rate/Reverse Data Rate = 44.9%

18. If the Forward Physical to Reverse Data rate is greater than approximately 63:1, verify the following RF data fill:

RL Max Rate Table

Recommended Num-AT-From Num-AT-To Rate

1 59 153.6 Kbps

RL Transition Probabilities Table

Attribute Recommended Transition009k6_019k2 128 Transition019k2_038k4 64 Transition038k4_076k8 32 Transition076k8_153k6 8 Transition153k6_076k8 255 Transition076k8_038k4 32 Transition038k4_019k2 16 Transition019k2_009k6 16

19. RL SHO failures can impact FL throughput by limiting the ACK rate on the RL. To determine the per UATI RL SHO failure rate, from CLI go to enable mode and view the per UATI connection counters:

1xEVDO session <UATI> counters <All>: Reverse Link Soft Handoffs :- Attempts : 0 Success : 0


Blocked by RN : 0 Blocked by RNC : 0 Failed by RN : 0 Failed by RNC : 0 Failed because of no TCC : 0

In EVDO soft or softer handoffs are initiated by the AT sending a Route update message to request the addition or removal of a pilot to its Active Set. Each Route update is counted as a handoff attempt. Soft handoff failures generally occur as a result of resource blocking or failures at the RNC or RN. Currently there are no reason codes for specific resource blocks.

Other failures may have occurred as a result of configuration errors. In the event of failures verify the following RF data fill parameters:

TCC Wait Timer

The amount of time the RNC waits for a traffic channel complete (TCC) message from the AT. A Traffic Channel Assignment message is typically sent by the RNC to the AT as part of a reverse link soft handoff. If the TCC is not received in time the connection will be closed.


1000 – 30000 1/1000 seconds

5000

IS856Neighbor

Defines the neighbors surrounding the sector an AT may be connected to. Inconsistencies in the neighborlist may result in missed handoffs. Insure that reciprocal neighbors are not missing in adjacent sites neighborlists.

19. If the Fwd Physical Rate to reverse data rate ratio is acceptable, a verification of the RLP Timeout rate should be performed. Based on an access layer PER of 1% the system will inherently encounter NAK timeouts (i.e. RLP packet retransmission is not received in .5ms or received in error) but the percentage should be small. If the percentage is large then there will be failures or retransmissions at the TCP layer. It should be noted that the relationship between PER of the initial packet and the PER of a retransmitted packet in effect makes the worst case PER .01% (i.e. PER initial =1% x PER for retransmitted =1% = .01%)

By example, assume that a 5Mbyte file (40Mbits) is being downloaded. If this is application layer we need to assume roughly 15% overhead (40Mbits *1.15 = 46Mbits). Each FTC MAC packet is 1024 bits so the system will send ~ 45,922 packets. Then the number of NAK timeouts that should be expected = 45,922x.01% = ~5NAK timeouts (i.e. packets not received or received in error on retransmission).


Example: RLP Packet Error Rate

Formula: Equivalent MAC packets = ((29724213Bytes x 8)/1024 = 232220 packets

Formula: RLP Packet Error Rate = SQRT (29 Packet Errors (NAK Timeouts) / 232220 packets) x 100% = 1.12%

20. In the event that RLP Packet Error Rate is >5% DOM logs should be collected to verify whether or not packets are being delayed at the DOM for resequencing. The following script can be used to log the GRE / ABIS packets seen at the DOM. Be careful as these logs are intrusive.

DOM_Logging.txt (1 KB)

21. If the celldm logs show packets are being received out of sequence at the DOM and there are excessive RLP NAKS as a result, the SDU Resequencing Wait Timer may be too short. The default wait period is 25milliseconds and is adjustable in increments of 5 milliseconds. The time may be updated with following CLI command:

NORTEL-XX(config)#1xevdo sdu-resequencing-wait <period> 22. To verify of further impact on throughput, TCP trace data should be collected using Windump or

Ethereal in parallel with AT logging. TCP traces will show whether or not there were delays in application throughput due to TCP segment losses / retransmissions.

23. TCP segment retransmissions are common when IP header compression is utilized in wireless networks. It is known that TCP has a slow recovery time when a TCP segment is lost or corrupted as the decompressor becomes unsynchronized. The decompressor only knows the next packet sequence it is supposed to receive by validating the checksum value calculated based on difference values from one header to the next. In the case of high speed wireless links, the contents of the TX buffer window (64k) will be exhausted before TCP recovers. All TCP packets will be discarded without Acknowledgment by the receiver before TCP sends a packet with an uncompressed header. The packet with the uncompressed header will allow TCP to recover by sending retransmissions of the discarded packets.


Example: TCP segment loss

Example: Time Sequence Graph (TCP packet throughput)

Compressor E D C B A RF Link E D B A Decompressor D C B A

This segment is really D not “C”

This segment is really E, not “D”

Packet loss or corruption over lossy medium

Compressor E D C B A RF Link E D B A Decompressor D C B A

This segment is really D not “C”

This segment is really E, not “D”

Packet loss or corruption over lossy medium


Example: Packet Loss

24. If many instances of stalled throughput or packet discards are seen in TCP traces, process the logs with TCPTrace http://tcptrace.org/. Use the (tcptrace-lr) command to determine what node is most affecting packet loss and / or latency.

Example: TCPTrace Output

================================

TCP connection 6:

host k: h226.13.39.162.ip.alltel.net:20

host l: 216.96.112.225:1112

complete conn: yes

first packet: Tue May 24 13:15:15.380032 2005

last packet: Tue May 24 13:23:39.204496 2005

elapsed time: 0:08:23.824464

total packets: 64180

filename: may24_16_eth

k->l: l->k:

total packets: 38734 total packets: 25446

ack pkts sent: 38733 ack pkts sent: 25446

pure acks sent: 2 pure acks sent: 25444

sack pkts sent: 0 sack pkts sent: 236

dsack pkts sent: 0 dsack pkts sent: 0

max sack blks/ack: 0 max sack blks/ack: 1

Packets are discarded.


unique bytes sent: 54888890 unique bytes sent: 0

actual data pkts: 38731 actual data pkts: 0

actual data bytes: 55305445 actual data bytes: 0

rexmt data pkts: 293 rexmt data pkts: 0

rexmt data bytes: 416556 rexmt data bytes: 0

zwnd probe pkts: 0 zwnd probe pkts: 0

zwnd probe bytes: 0 zwnd probe bytes: 0

outoforder pkts: 0 outoforder pkts: 0

pushed data pkts: 843 pushed data pkts: 0

SYN/FIN pkts sent: 1/2 SYN/FIN pkts sent: 1/1

req sack: Y req sack: Y

sacks sent: 0 sacks sent: 236

urgent data pkts: 0 pkts urgent data pkts: 0 pkts

urgent data bytes: 0 bytes urgent data bytes: 0 bytes

mss requested: 1428 bytes mss requested: 1460 bytes

max segm size: 1428 bytes max segm size: 0 bytes

min segm size: 452 bytes min segm size: 0 bytes

avg segm size: 1427 bytes avg segm size: 0 bytes

max win adv: 17136 bytes max win adv: 65535 bytes

min win adv: 16384 bytes min win adv: 2703 bytes

zero win adv: 0 times zero win adv: 0 times

avg win adv: 17135 bytes avg win adv: 65472 bytes

initial window: 2856 bytes initial window: 0 bytes

initial window: 2 pkts initial window: 0 pkts

ttl stream length: 54888890 bytes ttl stream length: 0 bytes

missed data: 0 bytes missed data: 0 bytes

truncated data: 0 bytes truncated data: 0 bytes

truncated packets: 0 pkts truncated packets: 0 pkts

data xmit time: 503.634 secs data xmit time: 0.000 secs

idletime max: 18076.0 ms idletime max: 17915.8 ms

throughput: 108944 Bps throughput: 0 Bps

RTT samples: 24962 RTT samples: 2

RTT min: 10.0 ms RTT min: 130.2 ms

RTT max: 200.3 ms RTT max: 140.2 ms

RTT avg: 5.8 ms RTT avg: 135.2 ms

RTT stdev: 12.1 ms RTT stdev: 0.0 ms

RTT from 3WHS: 0.0 ms RTT from 3WHS: 130.2 ms


RTT full_sz smpls: 24959 RTT full_sz smpls: 1

RTT full_sz min: 10.0 ms RTT full_sz min: 140.2 ms

RTT full_sz max: 200.3 ms RTT full_sz max: 140.2 ms

RTT full_sz avg: 5.8 ms RTT full_sz avg: 140.2 ms

RTT full_sz stdev: 12.1 ms RTT full_sz stdev: 0.0 ms

post-loss acks: 9 post-loss acks: 0

For the following 5 RTT statistics, only ACKs for

multiply-transmitted segments (ambiguous ACKs) were

considered. Times are taken from the last instance

of a segment.

ambiguous acks: 48 ambiguous acks: 0

RTT min (last): 0.0 ms RTT min (last): 0.0 ms

RTT max (last): 190.3 ms RTT max (last): 0.0 ms

RTT avg (last): 11.5 ms RTT avg (last): 0.0 ms

RTT sdv (last): 35.0 ms RTT sdv (last): 0.0 ms

segs cum acked: 13420 segs cum acked: 0

duplicate acks: 408 duplicate acks: 0

triple dupacks: 18 triple dupacks: 0

max # retrans: 2 max # retrans: 0

min retr time: 190.3 ms min retr time: 0.0 ms

max retr time: 3014.3 ms max retr time: 0.0 ms

avg retr time: 1258.5 ms avg retr time: 0.0 ms

sdv retr time: 856.2 ms sdv retr time: 0.0 ms

25. Throughput problems related to packet loss issues are often related to T1 connectivity of timing issues in the backhaul that result in path or line code violations. To see whether path or line code violations are consistently occurring, monitor the link status alarms in the DO EMS

27. Verify if the T1 timing source is incorrect or faulty T1 Timing Source. Set the interface to the correct timing source. Insure interfaces at both ends are the same.

NORTEL-XX>enable NORTEL-XX#configure NORTEL-XX(config)#controller t1 1/0/1 NORTEL-XX(config-controller)#clock source line-loop

Confirm timing configuration change: NORTEL-XX#clear counters NORTEL-XX#show controller t1 1/0/1

26. If packet loss is found from the TCP trace logs to be greater than 1%, a system health check should be performed as given in [17] page 30.


10.11 FL Per-Sector Throughput

10.11.1 Data Collection Methods 10.11.1.1 AT Logs The FL per sector throughput can only be determined from the AT logs when collecting logs from a stationary location. There is no good way to collect the per sector throughput from cluster drive data.

Method of Collection:

1. To collect data to measure FL per sector throughput from AT logs will require at least 2 ATs.

2. Configure all the ATs for HDR only operation with receive diversity enabled.

3. Insure IP header and Software Compression algorithms are disabled.

4.Testing should be performed when there is no other traffic on the sector.

5.Configure Autocall in Optis-M or Invex3G for the ATs to perform a single download of a 50MByte test file.

6.Apply a full EVDO log mask prior to starting collection.

7.Go to a location within the sector where average DRC request rate is between (1.2288 and 2.457Mbps) and C/I is consistently between 10 and 12dB.

8.Start download. When download is complete, stop logs and post process logs using EVDO RFO.

10.11.1.2 RNC Logs There is currently no useful information in the RNC logs that can be used to determine per Sector throughput.

10.11.1.3 OMs The aggregated sector throughput at the physical layer can be estimated using the OM PerfFLThruput. This OM only gives a time averaged estimate of the sector throughput based on a sliding 10 second window.


1. Telnet into the DOM and enable the FTC Scheduler. It is necessary to enter configuration mode to enable FTC scheduler NORTEL-03> config NORTEL-03 celldm scheduler ftc on

2. Log out of the DOM NORTEL-03> exit.

3. Repeat this process for all DOMs

4. The Average FTC Physical Layer throughput loading = perfFLThruput / 1000 (Unit: kbps)

5. Note: The throughput value given will only be for the last reported sampling window

6. A histogram of slot utilization and throughput can be generated using CLI show celldm flthruput <sector>

Sample: flthruput histogram from celldm

NORTEL-XX#show celldm flthruput alpha RATE: 38k4 76k8 153k6 307k2 614k4 921k6 1228k8 1843k2 2457k6 SLOTS: 0 0 0 0 0 0 0 0 159235 FL data throughput for sector alpha: 534749 bps


10.11.2 KPI Calculation 10.11.2.1 KPI Value and Tolerance Factor Forward link sector throughput is dependent upon the data rates being requested and the number of users on the sector.

10.11.2.2 Calculating KPI from AT Logs To determine the aggregated sector throughput from the AT logs go to RFO Statistics View, Forward Rate Statistics “Avg. ActualData Rate” this will be the average physical layer data rate of the sector.

Example: Average Actual Data Rate for all users (Physical layer sector throughput)

10.11.2.3 Calculating KPI from RNC Logs There are no sector throughput attributes currently available from RNC logs

10.11.2.4 Calculating KPI from OMs The aggregate sector throughput is as given in the celldm logs as shown in 10.11.1.3 above.

10.11.3 Troubleshooting Guide 1. In the event that there are problems with per sector data rate capability follow the steps in

“Stationary Troubleshooting” in Appendix A. .

2. Refer to section 10.10.3 for tips on troubleshooting sector throughput.

For a non commercial network sector throughput will not be a key indicator of performance problems due to low traffic density. In this case per user throughput will be a better indicator of potential problems.

For an in service network, sector throughput should be trended over time and records kept of the variability in performance. In the event that the average sector throughput dips below a typical value, based on trends, then the troubleshooting guidelines in Appendix A should be followed to resolve.

10.12 RL Per-User Throughput In the EVDO system the AN reports the status of the reverse link loading to the ATs by setting the Reverse Activity Bit (RAB). The AT uses the RAB and several other attributes to determine its reverse link data rate. Simply stated the AT reduces its data rate on the reverse link when loading is heavy and increases it when loading is light.



1. Use Optis-M or Invex3G to collect reverse link user throughput logs.

2. Configure the AutoCall or Data Connection tabs to perform an FTP PUT of a 5MByte file to a server in close proximity to the PDSN.

3. Configure the AT for HDR only operation with received diversity enabled.

4. Insure that all compression algorithms (S/W, IP header, etc) are disabled prior to collecting any AT log data.

5. Begin stationary tests or cluster drive test.


6. At the end of the test stop AT logs and post process using EVDO RFO.

10.12.1.2 RNC Logs There are no RL per user throughput attributes available in the RNC logs.

10.12.1.3 OMs The Reverse link physical layer throughput is estimated over a 3 second sliding window and is available from the per sector RL throughput OM “perfRLThruput”. The per user throughput can be estimated as follows:


1. Collect the Reverse Link histogram from the celldm logs show celldm rlthruput <sector>.

NORTEL-XX#show celldm rlthruput alpha RATE: 9k6 19k2 38k4 76k8 153k6 BYTES: 139648 533632 428928 38656 512 RL data throughput for sector alpha: 26112 bps


RL - ~ 80 kbps {depends on TP values and 1xRTT tune-away times (tune-away times should be less than 200 ms 95% of the time)}

10.12.2.2 Calculating KPI from AT Logs 1. After post processing the AT logs with RFO, open Statistics View Data Statistics Application

“TX Active Throughput”.

Example: RFO RL throughput statistics

10.12.2.3 Calculating KPI from RNC Logs There are no RL throughput attributes available in the RNC logs.

10.12.2.4 Calculating KPI from OMs 1. Calculate the weighted average reverse rate from the celldm data as follows:

∑=AllRates

BytesceivedTotalBytes )8*(Re

∑=AllRates

RateBytesemissionTimTotatTrans )/)8*((

emissionTimTotalTransceivedTotalBytesverseRateerageWeightedAv /ReRe =

torsfromallverseRateerageWeightedAvAverageofeerBurstRathysicalLayverseLinkP

sec)Re(Re =

2. The calculated average given above reflects the average burst rate per user. However, this is only an estimate and does not reflect the user experience exactly.


10.12.3 Troubleshooting Guide 1. For a point by point procedure for testing and troubleshooting, refer to Appendix A.

2. If a problem with Reverse Link Throughput is suspected, process the AT logs and verify that the Graph View that the Reverse Data Rate and the App TX Rate track closely. Also verify that the AT transmit power is not excessive.

Example: Graph View of Reverse Data Rate, App TX Rate, and TX Total Power

3. Verify that the laptop / collection tool TCP/IP settings are in accordance with Appendix E “TCP/IP Configuration”.

4. Verify if the problem is isolated to a specific DOM or DOMs. If the problem is over several DOMs troubleshoot Ethernet Performance or GRE / A10 resequencing. See page 4-12 and 4-75 in [18].

5. If the problem is isolated to 1 DOM then troubleshoot the site for Radio malfunction, antenna problems, feeder problems, DOM chassis malfunctions, or interference. See page 5-126 [18].

6. If the problem is random throughout the network troubleshoot the backhaul timing source or connectivity. See page 4.34 – 36 [18]

7. Verify the following RF datafill parameters are within recommended values established by Nortel and the customer:

Reverse Link Max Rate Table

This table provides a per sector datafill method to control the maximum data rate granted to the AT. CLI must be used to make changes to this table. Changes made to this table are not persistent (i.e. any restart of the DOM will cause the operator to have to go in and reset the values.

Recommended Num-AT-From Num-AT-To Rate

1 59 153.6 Kbps

RAB Threshold


0 – 16384 1/16384 % of Load

9830 (60%)


Reverse Traffic Channel Transition Probabilities Table

Whenever in a connection, the AT will always be transmitting at a particular data rate on the reverse link. Initially, the AT will always begin with a 9.6 Kbps data rate. But the next and successive frames will transmit at a rate determined by the Reverse Link Transition Probabilities listed above. The AT will increase, decrease or not change the data rate of the next transmitted frame depending on the data rate of the current frame, the state of the combined busy bit, and an internal random number generator ranging between 0 and 1.

Attribute Recommended Transition009k6_019k2 128 Transition019k2_038k4 64 Transition038k4_076k8 32 Transition076k8_153k6 8 Transition153k6_076k8 255 Transition076k8_038k4 32 Transition038k4_019k2 16 Transition019k2_009k6 16

8. RL SHO failures can impact the RL throughput. From CLI go to enable mode and view the per UATI connection counters to determine the number RL SHO failures:

1xEVDO session <UATI> counters <All>: Reverse Link Soft Handoffs :- Attempts : 0 Success : 0 Blocked by RN : 0 Blocked by RNC : 0 Failed by RN : 0 Failed by RNC : 0 Failed because of no TCC : 0 In EVDO soft or softer handoffs are initiated by the AT sending a Route update message to request the addition or removal of a pilot to its Active Set. Each Route update is counted as a handoff attempt. Soft handoff failures generally occur as a result of resource blocking or failures at the RNC or RN. Currently there are no reason codes for resource blocking. Other failures may have occurred as a result of configuration errors. In the event of failures verify the following RF data fill parameters:

TCC Wait Timer

The amount of time the RNC waits for a traffic channel complete (TCC) message from the AT. A Traffic Channel Assignment message is typically sent by the RNC to the AT as part of a reverse link soft handoff. If the TCC is not received in time the connection will be closed.


1000 – 30000 1/1000 seconds

5000

IS856Neighbor

Defines the neighbors surrounding the sector an AT may be connected to. Inconsistencies in the neighborlist may result in missed handoffs. Insure that reciprocal neighbors are not missing in adjacent sites neighborlists.

9. If problems persist perform a FTAP and RTAP tests.


10.13 RL Per-Sector Throughput

10.13.1 Data Collection Methods 10.13.1.1 AT Logs Method of Collection: 1. To measure sector throughput from AT logs testing will have to be performed in stationary locations

where RL sector throughput is known to be compromised. 2. At least 2 ATs will be required to measure the sector throughput from AT logs. 3. Configure the ATs for HDR only operation with receive diversity enabled. 4. Insure all compression algorithms are disabled prior to beginning the test. 5. Drive to a predetermined location (sector) within the RNC where the C/I is between (10 – 12dB). 6. Insure that the AutoCall setup in Optis-M is set for synchronous logging to insure all ATs begin

uploads at the same time. 7. Start AT data collection logs in each drive van. 8. Perform 5 consecutive FTP uploads of a 1MByte file for all ATs. 9. When logging is complete stop logs and use EVDO RFO to post process.

10.13.1.2 RNC Logs There are no sector throughput attributes available from RNC logs.

10.13.1.3 OMs The Reverse link physical layer throughput is estimated over a 3 second sliding window and is available from the per sector RL throughput OM “perfRLThruput”. The per sector reverse link physical layer throughput is available as a per sector OM.


1. The reverse link physical layer throughput can also be collected using the celldm logs.




Aggregated reverse link sector throughput is approximately 200 kbps per Qualcomm.

10.13.2.2 Calculating KPI from AT Logs 1. The aggregated reverse link throughput will be the summation of the average application or physical

layer throughput of each AT averaged over the number of ATs used.

rofATsTotalNumberoughputTXActiveTh

ughputSectorThroAggregatRLofAT∑=

#

1)(

10.13.2.3 Calculating KPI from RNC Logs Not Applicable

10.13.2.4 Calculating KPI from OMs 1. The aggregate RL physical layer sector throughput can be determined directly from the celldm logs.




10.13.3 Troubleshooting Guide Follow the troubleshooting guidelines for Reverse Link User throughput (7.12.3).

10.14 FL Achievable Data Rate Forward Link Achievable Date Rate is the maximum data rate that can be obtained by a user in ideal RF conditions anywhere within the network. The maximum achievable rate is basically the golden value for per user forward link throughput.


1. Follow the system integrity pre check guidelines given in the flowchart shown in Appendix A. prior to starting testing.

2. Insure that the AT is in HDR only mode and that receiver diversity is enabled.

3. Verify all compression algorithms are disabled.

4. Insure that there is a server located in close proximity to the PDSN for testing and troubleshooting purposes.

5. Configure Optis-M or Invex3G to perform 10 consecutive FTP downloads of a 5MByte test file.

6. Go to a location where C/I is between (10 – 12dB) and the DRC requests are between (1.2288 – 2.457Mbps).

7. Verify from the “User Size” attribute in Optis-M or XCAL that there are no other users attached to the sector.

8. Begin downloads. When completed, stop logs and post process with EVDO RFO.

10.14.1.2 RNC Logs Currently, there are not throughput attributes available from the RNC logs.

10.14.1.3 OMs There are currently no per user throughput OMs that can be used for determining the achievable data rate on the forward link. Achievable FL throughput can be estimated from the celldm logs show celldm flthruput <sector>


1. Telnet into the DOM and enable the FTC Scheduler. It is necessary to enter configuration mode to enable FTC scheduler NORTEL-03> config NORTEL-03 celldm scheduler ftc on

2. Log out of the DOM NORTEL-03> exit.

3. Repeat this process for all DOMs

4. The Average FTC Physical Layer throughput loading = perfFLThruput / 1000 (Unit: kbps)

5. Note: The throughput value given will only be for the last reported sampling window


6. A histogram of slot utilization and throughput can be generated using CLI show celldm flthruput <sector>

Sample: flthruput histogram from celldm

NORTEL-XX#show celldm flthruput alpha RATE: 38k4 76k8 153k6 307k2 614k4 921k6 1228k8 1843k2 2457k6 SLOTS: 0 0 0 0 0 0 0 0 159235 FL data throughput for sector alpha: 534749 bps

10.14.2 KPI Calculation 10.14.2.1 KPI Value and Tolerance Factor Maximum Achievable rate depends on the RF channel conditions in which the AT is embedded. The data rate is proportional to the DRC request rate ADR = αDRC, where 0<α<1.

10.14.2.2 Calculating KPI from AT Logs 1. After the data is post processed with EVDO RFO, open the Statistics View, Data Statistics,

Application ”RX Active Throughput”.

2. For maximum FL physical layer throughput go to, Statistics, Data Statistics, Forward Rate “Avg. Actual Data Rate”

10.14.2.3 Calculating KPI from RNC Logs No FL throughput data is available from RNC logs.

10.14.2.4 Calculating KPI from OMs 1. To approximate the Fl achievable throughput based on user experience, use the following method

based on the data in the celldm logs.

∑∑

=

AllRates

AllRatesSlots

RatexSlotsrdRateerageForwaWeightedAv

)(

10.14.3 Troubleshooting Guide 1. Refer to the troubleshooting flowcharts in Appendix A for a point by point process for testing a problem

resolution.

2. From the AT logs plot in the Graph View of RFO the RF characteristics (C/I, Avg. Effective DRC Rate, and Traffic Channel (TC) PER, seen at the location or locations where data was collected.


Example: Graph View of RF Parameters from RFO

2. If the average application layer throughput is low (e.g. < 1.5Mbps) where RF conditions are ideal (e.g. C/I

is in the 10-12dB range) perform a sanity check by performing a DOS FTP of the file. If the throughput is good (i.e. 1.5Mbps or greater) the check to insure that all compression algorithms that may be selectable in the collection tool are disabled.

3. Verify from the Sector Parameters Message in the AT logs that the correct Pilot PN, Band Class, Channel Number and Subnet Mask for the sector are correct.

4. Us DrTCP to insure that the TCP RX Window size is = 64240 (TCP Window Size in Windows Registry) and the MTU size = 1500Bytes (ProtocolMTU in Windows registry). Refer to Appendix E for more tips on TCP IP configuration.

5. If the throughput from the collection tool and DOS FTP are low verify that there are no other users on the site. In Optis or XCAL view the “User Size” in the EVDO Signal Graph or from the Quick Config message in the Layer 3 messaging available in RFO.

Example: User Size in Optis-M EVDO Signal Graph

6. If the User Size is equal to 1 verify that the DRC request rate is between 1.2288 and 2.457Mbps.

7. If the DRC request rate is low verify from the AT logs that the Fwd Physical rate to Reverse data rate ratio is approximately 40:1.


8. In the event that the Fwd Physical to Rvs Data rate is much greater than ~65:1 perform the following checks:

o Do the AT logs indicate drops and access failures?

o Is the RL SHO factor > 2? Show 1xEVDO connection <UATI> counters <all>

o Verify that there are no failures due to resource allocation failures? show 1xevdo connection <UATI> counters <all>

o Verify the RF datafill for RL Max Rate Table, and the Reverse Transition probabilities table See [8] for Nortel recommended RF datafill values and methods for updating datafill tables.

o Perform FTAP and RTAP tests to verify the physical connection between the AT and RNC.

Example: Fwd Physical to Reverse Data rate ratio

Formula: Fwd Physical Rate / Rvs Data Rate = 45.6

9. If the Forward to Reverse data rate ratio is acceptable, perform a health check of the backhaul links to verify that there are not physical errors being reported (i.e. path line code violations, etc). Perform a system health check (T1 interface status, Ethernet, etc) according to the User performance troubleshooting flow chart in [18].

10. T1 errors will result in reduced throughput, Abis failure, link status alarms, or wilted radios. Ethernet problems are characterized by discards or collisions and may be a result of duplex mode or speed configuration issues. Follow recovery steps in [18].

11. Collect TCP trace logs using Ethereal or Windump while performing a DOS or XCAL-DO FTP download. Use the Ethereal TCP sequence graph or TCPTrace extended output to determine is there is excessive packet loss. A TCPtrace analysis (tcptrace-lr) will indicate at what node packet loss or retransmissions are occurring as well as the average RTT time between nodes. This information can aid engineers in determining the source of problems.


Example: Ethereal TCP Sequence Graph:

12. An inspection of the RLP statistics in the AT logs will indicate NAK timeout (NAK timeout occurs

when an RLP packet is retransmitted in error or not received by the AT within the .5msec wait time) rate. If the NAK timeout rate exceeds 5 %, the SDU Resequencing Wait Timer may be too short.

13. To adjust the SDU Resequence Wait Timer use the following CLI command: NORTEL-XX# (config)#1xevdo sdu-resequencing-wait <period>

Example: RLP NAK Timeout Rate

Dropped TCP Packets TCP Packet

Retransmission


Formula: ((RX Total Bytes x 8) x 1.15)/1024 = RX Total packets (SQRT (NAK Timeouts / RX Total Packets)) x100 = NAK Timeout Rate

14. Perform FTAP and RTAP tests to validate the integrity of the physical link between the RNC and the AT [17].

10.15 RL Achievable Data Rate The Achievable rate for the reverse link represents the maximum data rate achievable from the network / sector given that all equipment is configured and operating as designed and the RF conditions are ideal. In ideal RF conditions the achievable rate will represent a golden value for the system.


1. Follow the system integrity pre check guidelines given in the flowchart shown in Appendix A prior to starting testing.

2. Insure that the AT is in HDR only mode and that receiver diversity is enabled.

3. Verify all compression algorithms are disabled.

4. Insure that there is a server located in close proximity to the PDSN for testing and troubleshooting purposes.

5. Configure Optis-M or Invex3G to perform 10 consecutive FTP uploads (PUT) of a 1MByte test file.

6. Go to a location where C/I is between (10 – 12dB) and the DRC requests are between (1.2288 – 2.457Mbps).

7. Verify from the “User Size or RPC Size” attribute in Optis-M or Invex3G that there are no other users attached to the sector.

8. Begin uploads. When completed stop logs and post process with EVDO RFO.

10.15.1.2 RNC Logs There are currently no useable attributes in the RNC logs that can be used to determine RL Achievable data rate.

10.15.1.3 OMs The Reverse link physical layer throughput is estimated over a 3 second sliding window and is available from the per sector RL throughput OM “perfRLThruput”. The per user throughput can be estimated as follows:




10.15.2 KPI Calculation 10.15.2.1 KPI Value and Tolerance Factor Maximum achievable throughput on the reverse link is contingent upon current sector loading conditions as well as the RF conditions seen at the collection location. In ideal RF conditions the maximum achievable rate should be in the 100 – 140kbps range.



1. After post processing the AT logs with RFO, open Statistics View Data Statistics Application “TX Active Throughput”.

Example: RFO RL throughput statistics

2. To determine the maximum RL throughput at the physical layer in RFO, Statistics View Data Statistics Reverse Rate “Avg. Data Rate”.

Example: RFO RL Physical Layer Statistics

10.15.2.3 Calculating KPI from RNC Logs There are no RL throughput attributes available from the RNC logs.

10.15.2.4 Calculating KPI from OMs 1. To determine the maximum achievable rate from OMs, calculate the weighted average reverse rate

from the celldm data as follows:

∑=AllRates

BytesceivedTotalBytes )8*(Re

∑=AllRates

RateBytesemissionTimTotatTrans )/)8*((

emissionTimTotalTransceivedTotalBytesverseRateerageWeightedAv /ReRe =

torsfromallverseRateerageWeightedAvAverageofeerBurstRathysicalLayverseLinkP

sec)Re(Re =

2. The calculated average given above reflects the average burst rate per user. However, this is only an estimate and may not reflect the maximum achievable RL data rate of the sector.

10.15.3 Troubleshooting Guide 1. Refer to Appendix A for the point by point testing and troubleshooting process.

2. For more detailed troubleshooting examples and references for RL performance, refer to section 7.12.3 “RL per User Throughput troubleshooting”.


10.16 Packet Latency

10.16.1 Data Collection Methods 10.16.1.1 AT Logs 1. Insure that the AT is in HDR only mode and receiver diversity enabled.

2. Insure all compression algorithms are disabled and TCP/IP settings are configured as per Appendix D.

3. The Ping utility in Optis can be used to send multiple pings with a user defined payload for the purpose of measuring packet latency across the network.

4. Pings should be sent to the PDSN IP address.

5. Configure Optis-M or Invex3G for Ping and Trace tests as shown below:

Example: Optis-M Autocall for Ping

6. An alternative to using Ping to measure packet latency is to collect TCP trace logs with Ethereal or

Windump.

7. Since XCAL and Invex3G-PC uses the PC’s protocol stack for data collection, Ethereal or Windump can be used concurrently to capture TCP trace data. In the event XCAL or Invex3G-PC are not available for use, the engineer can open an Ethereal or Windump log and subsequently perform a DOS FTP download or upload to the test server.

8. After the Ping test and trace logs are collected stop logs and use TCPTrace or the playback function in XCAL to determine the average packet latency.

10.16.1.2 RNC Logs RNC logs do not contain any useable attributes for measuring packet latency.

10.16.1.3 OMs There are no packet latency OMs available


10.16.2 KPI Calculation 10.16.2.1 KPI Value and Tolerance Factor For 1xEV-do system only {using PDSN to ping AT Latency shall be less than 200 ms, should be < 150 ms}. Systems Engineering times are 150 ms (mean), 172 ms (90th percentile).

10.16.2.2 Calculating KPI from AT Logs 1. To determine the KPI from the Ping test conducted with XCAL or Optis, use the replay function and

from the Status menu select “Ping Status” to display the latency over time. Also from the Status menu the TraceRT Status can be selected to show the average RTT per node.

2. Alternatively, the TCP trace logs collected can be processed using TCPTrace (tcptrace-lr) which produce packet latency statistics per node as shown in the example below:

Example: TCP Trace output

RTT samples: 24962 RTT samples: 2 RTT min: 10.0 ms RTT min: 130.2 ms RTT max: 200.3 ms RTT max: 140.2 ms RTT avg: 5.8 ms RTT avg: 135.2 ms RTT stdev: 12.1 ms RTT stdev: 0.0 ms

10.16.2.3 Calculating KPI from RNC Logs 10.16.2.4 Calculating KPI from OMs

10.16.3 Troubleshooting Guide 1. Ping, Pathping, TCPTrace, Ethereal, or other protocol analysis tools can be used to identify packet loss

or latency problems between the AT, PDSN or the application server.

2. Perform several FTP downloads while capturing TCP trace or Ethereal logs

3. Process the logs using TCP Trace with the “lr” option to determine the node or nodes where packet latency is excessive “RTT Ave”.

4. If the packet latency is >>300ms on any intermediate nodes follow the troubleshooting flowchart shown in Appendix A.

5. Packet latency can often be a result of: T1 path or line code errors or faults, DOM PPP interface configuration, Ethernet configuration, T1 aggregator problems port issues, TCP configuration, or server issues.

6. Methods to troubleshoot these issues are covered in [18].

10.17 DO-To-1x Handoff Delay


Method of Collection

1. Configure autocall in Optis to allow the AT to perform an FTP download of 50MB file.

2. Insure that keep PPP and allow dormant mode options are selected in the autocall setup.

3. Go to a location on the outskirts of the EVDO coverage area and start the AT logs.

4. Drive out of EVDO coverage but within the coverage of the underlying 1xRTT coverage area.

5. Perform multiple drives from within EVDO coverage into areas where EVDO coverage ends and where there is only 1xRTT coverage.

6. Stop and save logs after each pass.


7. Make note that the EVDO to 1xRTT handoff is actually an EVDO connection drop followed by an origination on 1xRTT. The FTP download application started on EVDO should continue on 1xRTT from where it stopped on the EVDO network.

10.17.1.2 RNC Logs There is currently no useful information regarding EVDO to 1xRTT handoffs in the RNC logs.

10.17.1.3 OMs There are no current OMs to show EVDO to 1xRTT handoff delay.


From [14]: 15s for SIP 25s for MIP

10.17.2.2 Calculating KPI from AT Logs 1. Using EVDO RFO, post process the log files collected for the EVDO to 1xRTT handoff test.

2. DO to 1x handoff delay is measured as the time delta between the Connection Release on EVDO and the Service Connection Complete message on 1xRTT.

3. Determine 90th percentile delay for all valid EVDO to 1x handoffs.


10.17.2.4 Calculating KPI from OMs N/A

10.17.3 Troubleshooting Guide EVDO to 1xRTT handoff is governed by the aggregate value of Ec/Io as determined from all EVDO pilots in the AT’s Active and Neighbor sets. Once the aggregate value of all of these drops below a vendor specific value, the AT will declare the EVDO system lost and attempt to originate on the 1x network. The time required to make a seamless transition is as given above. However, other factors may influence the time delay:

1. Intermittent recovery of EVDO coverage as influenced by terrain, multipath etc. In these cases the EVDO connection may be released by the AT due to poor coverage thus, loss of supervision, However, if supervision is recovered the AT may attempt to reconnect to EVDO prior to originating on 1x. In these cases little can be done to improve the transition performance.

2. Excessive delay can also be caused by the AT having to search the PRL for the 1x system. In this case the PRL should be optimized. The AT PRL can be optimized using the Qualcomm utility QPST.

3. Insure that the AT has been datafilled in the CDMA system HLR.

10.18 DO-To-1x Success Rate



Same as EVDO to 1x handoff delay

10.18.1.2 RNC Logs N/A


10.18.1.3 OMs N/A

10.18.2 KPI Calculation 10.18.2.1 KPI Value and Tolerance Factor 10.18.2.2 Calculating KPI from AT Logs

1. Using EVDO RFO post process the collected AT logs collected for EVDO to 1x delay / success rate testing.

2. Do to 1x handoff success rate is based on the number or 1x originations that result in a continuation of the application process.


10.18.2.4 Calculating KPI from OMs N/A

10.18.3 Troubleshooting Guide Same as above


Appendix A: EVDO Optimization and Troubleshooting Flow Charts

Optimization and Troubleshooting Flow Chart

Perform Network pre checks and preliminary parameter

audits Page 2

Is the network in a commercial or launched

state?

Perform site Shakedowns Perform Baseline Cluster Drive Tests

N Y

Do specific sites or areas have performance issues? Do specific sites or areas

have performance issues?

Perform Stationary troubleshooting tests

Page 3 (#2)

Perform mobility troubleshooting tests

Page 4 (#3)

N

Y

Are stationary perf and optimization objectives

complete?

Y

YN

Are mobility performance and optimization objectives

complete?

N

Complete Final Cluster Drives

Y

N

Y


Y

N

Y

N

N

NE operational status is Active? NORTEL-XX# show node

All modules have Active status? NORTEL-XX# show modules

N

Go to troubleshooting sections of NTP 411-2133-949

Can Ping the following from DOM / AT?

- Node IP of DO-RNC - IP of Agg Router interfaces for backhaul - IP of DO -EMS

Y

Administrative & operational status of all interfaces is Active? (PPP, LCP,

Ethernet) NORTEL-XX# show interfaces

Y

N

DO-RNC must be able to Ping: - Node IP of all connected DOMs - IP address of DOM PPP interfaces - IP address of DO-EMS - All PDSNs

N

Y

Is ABIS Peer status = Connected for all DOMs? NORTEL-XX# show abis peer

Y

N

Is GPS present and locked for DOMs? NORTEL-XX# show GPS health

Y

Is SNTP or system time correct / present? NORTEL-XX# show SNTP Time

N

System Integrity Verification Checklist

Y

Y

Is GRE A10 Resequencing enabled for all BIO modules?

NORTEL-XX (config) # fastpath a10 reseq 1

N

RF Datafill -Verify RF Datafill is in accordance with Nortel recommended values or what has been mutually agreed to with

Y

Refer to Nortel 1xEVDO RF Datafill Guide v 2.6

(March 2 2005)

N

Begin RF Optimization Tests (Stationary or Mobile)

TAP Tests (Tap Users Guide) (FTAP / RTAP) Passed

TCP/IP settings are according to recommended values (e.g. IP header Comp. off, MTU, etc)?

Y

Refer to Appendix E 1xEVDO Optimization Guide

N

N

Set up Data Collection tools according to:

EVDO Data Collection Tools – Optis-M and XCAL-DO v 1.1

Y


Go to site or location where problems are and perform FTP DL using data collection tool.

Y

Is Fwd. Physical Rate to Rvs. Data rate ratio ~ 40:1 but not > 63:1?

Y

N

Y

N

Check User Size. Is it greater than 1? >> 1 user on site, test where or when

there is less traffic

N

Y

Stationary RF Troubleshooting

Is Avg. C/I ~8 - 12dB? N

Y

Is Avg. App t’put ~ 1.5 – 2.0Mbps?

N

Go to next location or begin mobility testing

N

Y

Verify that there are no antenna feeder issues

Post Process AT logs with RFO

Do AT logs indicate >5% Drops or AF?

Refer to Chapter 7 of EVDO Opto guide for Drop and AF resolution guidelines.

Are DRC requests ~ 1.2288 - 2.457Mbps?

-Try a different AT - Verify AT COM ports are configured correctly?

Y

N

Y

Is DOS FTP t’put equivalent to what tool reports?

Data Collection Tool Settings -Verify BAUD rate in port settings is ~115200 -Verify phone model and chip set are correct -Verify correct PPP adapter under port settings

- Is RL SHO factor >2.0? show 1xevdo connection<uati> counters<all>

Is RL SHO failure due to resource allocation fail show 1xevdo connection<uati> counters<all>

Verify data fill for RL Max Rate and trans Probs

- Are there excessive RL NAKS/Timeouts in AT logs?

Opt coverage to intro dominant server

Determine resource issue and resolve

NY

Y

YN

N

Go to Page 5 (#3)

Go to Page 5 (#3)

N

Stationary Test Start -

#1

Verify iuplink t’put (FTP PUT) in good RF ~140kbps?

N

Are Results as expected?

Go to Page 5 (#3)

N

Y


Start Mobility Tests #2

Drive Cluster Routes performing consecutive FTP DLs.

Y- From NEWTUN identify DOMs with high fail rates. - From RFO and NEWTUN output, categorize failures (missing neighbor, border handoff, long tune away, Session setup fail, poor coverage, etc).

N

Collect AT & RNC logs

Y

N

Mobile RF Troubleshooting

Process AT and RNC logs using RFO and CROS tools

Is Avg. RX Active t’put ~ (500 – 800kbps or greater) per cluster?

END

From AT and RNC logs, are there excessive drops and access

failures (i.e. >10%)?

Refer to the EVDO Optimization Guide, Chapter 10 for tips on

recognizing and addressing failures.

From AT and RNC logs are there areas with poor coverage (C/I < -2dB, PER >1%, > 3 ASet Pilots)?

- If there are Aset pilots > 2 tiers, try adjusting the “Cell Radius” according to formula in section 2.3.1.5 of 1xEVDO RF Data fill Guide - Add sites or make adjustments in antenna configurations to introduce dominant server. - Verify that there are no antenna feeder problems.

Y

Is Fwd Physical Rate to Rvs. Data rate ratio ~ = 40:1 but not > 63:1?

- Is RL SHO factor >2.0? show 1xevdo connection<uati> counters<all>

RL SHO failure due to resource allocation fails? show 1xevdo connection<uati> counters<all>

Optimize coverage to introduce dominant server

Determine root cause of resource issues and resolve

N

N

N

N

Y

Y

For site specific issues, go to

Stationary Tests #1

Go to areas where ratio is compromised and verify max uplink rate via FTP PUT.

Verify data fill for RL Max Rate and RL trans Probabilities table “1xEVDO RF Dataill Guide”

N

Are results as expected?

Y

Y

Go to Page 3 #1

Identify localized areas with poor FL

t’put

Go to Page 5 (#3)


N

TCP Trace analysis - #3

TCP / UDP / RLP Troubleshooting

Do TCP Trace logs (tcptrace –lr) indicate > 1% retransmissions?

Y

Y

Go to: Page 2 (#1) or

Page 3 (#2)

Incorrect or faulty T1 Timing Source. Set the interface to the correct timing source. Insure interfaces at both ends are the same. NORTEL-XX>enable NORTEL-XX#configure NORTEL-XX(config)#controller t1 1/0/1 NORTEL-XX(config-controller)#clock source line-loop Confirm timing configuration change: NORTEL-XX#clear counters NORTEL-XX#show controller t1 1/0/1

From CellDM (DOM) logs for RTC (rtclogmask 0xFFFFFFFF) are GRE / Abis packets arriving out of order?

Verify TCP/IP Settings – See EVDO Opto Guide Appendix E

Are there consistent T1/E1 Path or line code violations? 1 Monitor Link Status alarms for the interface for a period of time in DO-EMS. 2 NORTEL-XX(config)#logging console on

SDU Resequencing Wait Timer may be too short. NORTEL-XX# (config)#1xevdo sdu-resequencing-wait <period>

- From FL AT logs is the Ratio of AT NAK Bytes Requested / RX Total Bytes >= 5%? - Are the number of NAK Timeouts >> ((RX Total bits x 1.15) x .01^2)

Use Ethereal or Win Dump to collect TCP Trace logs

Y

Verify which node or interface is problematic and troubleshoot.

See NTP411-2133-949

Is RTT excessive between any nodes (e.g. >>300ms)?

Y

N

N

Y

N

N


Appendix B: CLI System Readiness Check

This appendix gives an example of how to do a basic readiness checkup using the command line interface. In this example, the following addresses are used: DOM IP Address = 47.135.203.191 ANC IP Address = 47.135.203.161. PDSN IP Address = 47.135.203.234 FTP Server IP Address = 47.135.203.18 Note: A DOM with 2 T1s configuration is used in the example 1. Telnet to <DOM node> (i.e., specific DOM IP address) via either Console Port (Dial up connection) or

Backhaul if available. 2. Under DOM prompt (NORTEL-03), type: NORTEL-03>en NORTEL-03#show interface Name Connection IP Address In Octets Out Octets Operational ----------------------------------------------------------------------------------------------------- ethernet1/3/1 10.10.32.156/20 0 0 Down node1/3/1 47.135.203.191/32 0 0 Up ppp1/3/1 0.0.0.0/0 0 0 Down ppp1/3/2 10.11.2.2/30 452184005 340 Up ppp1/3/3 10.11.3.2/30 440974939 51272523 Up ppp1/3/4 0.0.0.0/0 0 0 Down t11/3/1 0.0.0.0/0 0 0 Down t11/3/2 0.0.0.0/0 452184005 330 Up t11/3/3 0.0.0.0/0 440974939 58087164 Up t11/3/4 0.0.0.0/0 0 0 Down ----------------------------------------------------------------------------------------------------- This displays all interface status from DOM. For instance, the DOM Ethernet port is not used, the DOM node is up, 2 physical links (ppp1/3/2 and ppp1/1/3/3) between DOM and router are up, 2 T1 ports are up, and 2 T1 ports are not used. This information indicates that the interfaces are up and running. This only indicates that the DOM side interfaces are up but not necessarily all interfaces running properly. For example, the show interface command may indicate that the TWO T1s are up, but in fact one of the T1s may not be carrying any traffic at all. We have to ensure that IP route table is properly filled, and also check if the displayed values of “In Octets Out Octets” are incrementing for all the T1s. (In Octets and Out Octets information is displayed using the show interface command in the enable mode).


NORTEL-03>show ip route IP ROUTING TABLE ---------------------------------------------------------------------------- Destination Flags Gateway Owner Interface ---------------------------------------------------------------------------- 0.0.0.0/0 10.23.1.1 S ppp1/0/1 10.23.2.1 S ppp1/0/2 10.25.3.1 S ppp1/0/3 10.25.4.1 S ppp1/0/4 10.23.1.0/30 L 10.23.1.2 L ppp1/0/1 10.23.2.0/30 L 10.23.2.2 L ppp1/0/2 10.25.3.0/30 L 10.25.3.2 L ppp1/0/3 10.25.4.0/30 L 10.25.4.2 L ppp1/0/4 47.135.201.245/32 L 47.135.201.245 L node1/0/1 3. Under DOM prompt, type “show abis peer” in the enable mode as the following example: NORTEL-03>en NORTEL-03#show abis peer --------------------------------------------------------------------- PeerAddress Port State RxMsgs Rxhellos 47.135.203.161 5604 UP 18189 8163 ---------------------------------------------------------------------- This displays the link status between the DOM and the ANC. 4. Under DOM prompt, type “show rrc state” in the enable mode as the following example: NORTEL-03>en NORTEL-03# show rrc state ----------------------------------------------------------------------------- MODEM 0: ADMIN STATE: OPERATING OPERATION STATE: UP ======================================= SECTOR 16: ADMIN STATE: UP OPERATION STATE: UP ======================================= SECTOR 17: ADMIN STATE: UP OPERATION STATE: UP ======================================= SECTOR 18: ADMIN STATE: UP OPERATION STATE: UP ------------------------------------------------------------------------------- This displays sector status of the DOM. For instance, each sector under this DOM is up and running. SECTOR 16 = Sector Alpha, SECTOR 17 = Sector Beta, and SECTOR 18 = Sector Gamma. We can also use “show sector-element” command in the enable mode to display similar information: NORTEL-03#show sector-element Name Carrier Sector CAI Channel PN Power Admin Oper ------------------------------------------------------------------------------- element1 carrier1 sector1 IS-856 1125 116 25 dBm UP UP element2 carrier1 sector2 IS-856 1125 120 25 dBm UP UP element3 carrier1 sector3 IS-856 1125 124 25 dBm UP UP


5. Under DOM prompt, type “show traffic-channel” as the following example: NORTEL-03>en NORTEL-03#show traffic-channel BTS BSC Flow Pri Dn Pkts Up Pkts ------------------------------------------------- 0x0010 0x000034d6 CCH 128 124 26 0x0011 0x000034d7 CCH 128 183 76 0x0012 0x000034d8 CCH 128 105 2 This information provides how many traffic channels are established between DOM and ANC. The provided Example displays only CCH (Control Channel) since there is no active traffic channel at the time the command “show traffic-channel” is used. 6. From the DOM prompt, ping ANC, PDSN (i.e. ISP), and FTP servers as the following example: ------------------------------------------------------------------------------------------ NORTEL-03>ping 47.135.203.161 Sending 5, 100-byte ICMP Echos to 47.135.203.161, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 0/6/10 ms NORTEL-03>ping 47.135.203.234 Sending 5, 100-byte ICMP Echos to 47.135.203.234, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 0/6/10 ms NORTEL-03>ping 47.135.203.18 Sending 5, 100-byte ICMP Echos to 47.135.203.18, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 0/8/10 ------------------------------------------------------------------------------------------- Repeat these steps for each DOM in DO network. FTAP and RTAP Tests: To insure the integrity of the data link between the AT and the RNC, FTAP and RTAP tests should be performed. These tests are similar to Markov calls in CDMA voice as it sends simulated data packets from the RNC to the AT to test the link between the two. The following document should be used as a guideline for FTAP and RTAP tests:

D:\Profiles\mwoodleyMy Documents\CDMA


Appendix C: Shakedown Form

(NEEDS TO BE UPDATED: IP ADDRESS, FTP INFO, ETC.) Date/Time: Site ID/AN# AT Log file Site Name Other Logs Zone ID AT Model & Version Address Data Collection Tool/Version Engineer Network Software Version Problems Resolved? Status: Database Updated: _____________ Problems Comments/Observation

Handoff between Sectors Alpha <-> Beta Alpha <-> Gamma Gamma <-> Alpha

ALPHA (X) BETA (Y) GAMMA (Z) Datafill Observed Datafill Observed Datafill Observed Pilot PN Pilot PN Pilot PN Height Height Height Orientation Orientation Orientation Mech. DT Mech. DT Mech. DT Antenna Antenna Antenna Neighbor List Neighbor List Neighbor List Datafill Observed Datafill Observed Datafill Observed Call Initiation/Term. Call Initiation/Term. Call Initiation/Term. Coverage/HO Coverage/HO Coverage/HO


Appendix D: Examples of Past Issues

Problem Description:

Dropped Connections due to long tune away: Symptoms:

Drops will occur when tuning away to 1x (hybrid mode) for periods longer than 5 seconds. When the AT tunes to 1x, it looks for the last PN that it was monitoring on 1x and then sees if there is a stronger pilot. If the PN is the same as the last time, or it has tuned to that PN within the last 10 minutes, the AT will look for a page, and if it sees no page for it, it goes back to EV-DO. If the AT does an idle handoff to a “new” PN, then it must wait and read all the overhead messaging for that new PN (system parameters, extended system parameters, neighbor list, channel list, access parameters). This will typically add about 1.2s to the tune-away time, for each new PN that it does idle handoff to. In an area with no dominant server, it may hand off to several PN’s, and since each new one adds 1.2s, it may drop the EV-DO call. Another possible cause of a long tune away can occur when the Ec/Io of the reference 1x pilot drops below -16dB. When this occurs the AT will perform a rescan of the entire neighbor list which may take some time if there is a long neighbor list, or no dominant pilot to lock onto. (i.e. areas with pilot pollution issues). It should be noted that tune-away problems are caused by the 1x system, and not the EV-DO system.

How to recognize a long tune away: Figure 1 – EVDO RFO graph of RF conditions during and after a long tune away


Note: RF conditions are good a start of tune away to 1x. The figure above shows a graph view from EVDO RFO of the RF characteristics before and after a long tune away. During a long tune away the AT‘s reported RX power and TX power will appear constant until the AT retunes to EVDO.

Figure 2 – CDMA RFO graph of RF conditions during a tune away


Figure 3 – CDMA RFO – Grid view of tune away On a prior tune away, the mobile saw PN 324 with Ec/Io just 2dB below PN 320. As a result,

on the next tune away the AT tuned to PN 324 where, at that time, the Ec/Io was very poor. Resolutions / Workarounds:

Insure neighbor reciprocity to reduce chances that mobile will idle handoff to a new PN but cannot get back to the original if it needs to.

Avoid having neighbor lists with sites 2 tiers away from the reference site. If a long tune away is the result of RF optimization needs on the 1x network, permission

from the customer will be required before going forward with any changes to the underlying 1x network.


Required Tools: EVDO RFO CDMA RFO XCAP-DO


Connection Deny received from RNC if AT sends Route Update while RNC is in an “Await TCC” state. This issue is more commonly referred to as an “RNC State mismatch”.

Symptom:

Typically occurs after the first access Attempt (Connection Request) following an AF in the AT logs. This occurs when the mobile does not receive the TCA due to tune away or because it was in poor coverage. The state mismatch will occur when a Route Update (Connection Request) is received when the RNC is in a CSMAwaitTCCState rather than the CSMDormantState. When the mismatch occurs the system will send a Connection Close with Reason Code 2 which equates to a “Connection Deny”. The Connection Deny means that system resources are currently unavailable.

Resolution:

In prior releases there was a 5 sec timer associated with the tear down of the MAC channel. In a recent patch to release 2.1, this timer has been removed to insure that Connection Closes happen more quickly. In addition to removing the MAC channel timer to improve connection close performance, the 2.1 release also has been improved such that if a new Connection Request is received while a previous Connection is not closed out, the system will release the prior connection and set up the new one without pegging an Access Failure.


Cannot exceed 1.5MBps on FL

Symptoms: In good RF conditions where C/I >=10dB forward link throughput does not exceed 1.5MBps.

Resolution or Workaround:

Insure that backhaul T1’s are bundled and not separate. If T1’s are not bundled, throughput will be limited to 1.544MBps.

Log into the backhaul aggregate router and insure that there are no Path line-code

errors on configured T1’s.

Insure that the TCP window size and MTU settings for data collection tool are set to the optimum configuration as discussed in the throughput troubleshooting section below:

Inusre IP header compression is enabled.


Problem Definition:

Dropped connection during connection close This condition is more commonly referred to as “All or Nothing”.(i.e. the connection is granted “all” or the connection is denied “nothing”.

Symptoms:

In some cases after an FTP session has completed and the system is in the process of closing the connection, the AT may send a handoff Route Update prior to receiving the Connection Close (Race Condition). In these situations the system may send the TCA for the Route Update prior to the Connection Close. If the AT receives the Connection Close prior to receiving the TCA it may close out the connection without sending a TCC. As such, the system closes the connection with an error instead of a normal connection. This pegs as a drop, with reason code 2.

Resolution: The system should ignore Route Updates received after a Connection Close has been sent

thus avoiding this problem. The patch (.21) to 2.1 that speeds up the Connection Close will also help prevent occurrences of this symptom.

1x evdo optimization_guide_v3.0_chr

Documents

Transcript of 1x evdo optimization_guide_v3.0_chr