pebscdd012924e0110

BSC & TCU e3 Upgrade and Build On Line Improvements

Document number: PE/BSC/DD/012924 Document issue: 01.10 / EN Document status: Standard Date: 10-May-06

External document

Copyright© 2004 Nortel Networks, All Rights Reserved

Printed in France

NORTEL NETWORKS CONFIDENTIAL:

The information contained in this document is the property of Nortel Networks. Except as specifically authorized in writing by Nortel Networks, the holder of this document shall keep the information contained herein confidential and shall protect same in whole or in part from disclosure and dissemination to third parties and use same for evaluation, operation and maintenance purposes only.

The content of this document is provided for information purposes only and is subject to modification. It does not constitute any representation or warranty from Nortel Networks as to the content or accuracy of the information contained herein, including but not limited to the suitability and performances of the product or its intended application.

The following are trademarks of Nortel Networks: *NORTEL NETWORKS, the NORTEL NETWORKS corporate logo, the NORTEL Globemark, UNIFIED NETWORKS. The information in this document is subject to change without notice. Nortel Networks assumes no responsibility for errors that might appear in this document.

All other brand and product names are trademarks or registered trademarks of their respective holders.


Nortel Networks confidential

PE/BSC/DD/012924 01.10 / EN Standard 10-May-06 /58

PUBLICATION HISTORY

08/September/2004

Issue 01.00 / EN, Draft

Creation

22/October/2004

Issue 01.01 / EN, Preliminary

Added IN/TCU critical path improvements

Changed upgrade status state change

Added a new TGE upgrade TCU activate

Added CP duplexity check during an online upgrade

26/October/2004


Added IN/TCU offline upgrade robustness improvements

Added upgrade performances comparisons with V15.1

10/Decembre/2004


Comments after review

Updated background flashing protocol that applies only on TMU1_TM & OMU1_TM

Added upgrade strategy V16 through an intermediate V15.1.1 load

21/Janvier/2005


Changes the IN/TCU upgrade strategy to V16 with minimal backward compatibility

Updates the TCU upgrade TGE and impact at MMI

11/Mars/2005


Modifies Upgrade offline and online data flow

Modified TGE ‘resume Upgrade’ structure

Introduce OMU online upgrade improvements




17/Mars/2005


After Review from OMC architects

Added reduced TMU SS7 timer values during TMU online upgrade

Modified 2035 ANO probable cause with permanent freeze state

Reviewed performances estimations based on latest V15.1 measurements

25/Mars/2005


After Review from Upgrade CN designers

Modified OMU online upgrade with two OMU restarts

Change SupCn responsibility to trigger upgrade state changes

7/April/2005

Issue 01.06 / EN, Standard

After Review from CNP designers

Clarify board and global upgrade states

BOARD_UPG_PERFORMED applies only to online upgrade

BOARD_UPG_NEEDED is set only to ENABLE/ONLINE boards

Clarify responsibility of Sup_IN/TCU/IN wrt to board upgrade states

Precise that ResumeUpgrade TGE may require a BDE rollback

18/April/2005


After Review with OMC designers

Precise that TGE resumeUpgrade is only available at MMI

Precise BSC recommendations in case of permanent freeze state

20/September/2005


Remove manual on line rebalancing section

Update OMU on-line Upgrade chronogram

16/November/2005





Correct timings in OMU on-line Upgrade chronogram

Add a section regarding “resumeUpgrade” TGE

Correct the probable causes list of 2035 Alarm

Clarification online ugrd: upgrade both SS7 TMUs last - passive then active

10/May/2006


Several clarifications are given regarding TCU offline upgrade as well as IN/TCU improvements.




CONTENTS

1. INTRODUCTION............................................................................................................................8 1.1. OBJECT....................................................................................................................................8 1.2. SCOPE OF THIS DOCUMENT .......................................................................................................8 1.3. AUDIENCE FOR THIS DOCUMENT ................................................................................................9

2. RELATED DOCUMENTS ............................................................................................................10 2.1. APPLICABLE DOCUMENTS ........................................................................................................10 2.2. REFERENCE DOCUMENTS........................................................................................................10

3. BENEFITS....................................................................................................................................11 3.1. CUSTOMER/SERVICE PROVIDER BENEFITS................................................................................11 3.2. END-USER/SUBSCRIBER BENEFITS ........................................................................................11

4. FUNCTIONAL DESCRIPTION ....................................................................................................12 4.1. TCU OFFLINE UPGRADE ..........................................................................................................12 4.2. BSC OFFLINE UPGRADE PRINCIPLE .........................................................................................14 4.3. BSC START-UP .....................................................................................................................15 4.4. BACKGROUND FLASHING OF TRANSITION MODULES .................................................................18 4.5. ACCURATE UPGRADE STATUS........................................................................................19 4.6. UPGRADE ONLINE ENHANCEMENTS.........................................................................................24

4.6.1 TMU UPGRADE ENHAncED PROTOCOL ..................................................................24 4.6.2 Permanent Freeze State ...............................................................................................26 4.6.3 ResumeUpgrade TGE...................................................................................................26 4.6.4 RedUCED TMU SS7 PROBATION TIMER ..................................................................27 4.6.5 Improved OMU upgrade online.....................................................................................28

4.7. SPECIFICATION .......................................................................................................................29 4.7.1 IN/TCU Critical Path Improvements..............................................................................29 4.7.2 IN/TCU Offline Upgrade Robustness............................................................................29 4.7.3 BACKGROUND FLASHING .........................................................................................35

4.8. FEATURE INTERWORKING ........................................................................................................39 4.9. PERFORMANCES.....................................................................................................................39

5. INTERFACES ..............................................................................................................................40 5.1. GSM INTERFACES ..................................................................................................................40

5.1.1 AIR interface..................................................................................................................40 5.1.2 A interface .....................................................................................................................40 5.1.3 Lb interface....................................................................................................................40 5.1.4 Abis interface ................................................................................................................40 5.1.5 Ater interface.................................................................................................................40




5.1.6 AGPRS Interface...........................................................................................................40 5.1.7 Gb Interface...................................................................................................................40

5.2. O&M INTERFACES ..................................................................................................................41 5.2.1 Local Manager (Man Machine Interface) ......................................................................41 5.2.2 External Manager (Q3 / 3GPP interface) ......................................................................45 5.2.3 SDO...............................................................................................................................46 5.2.4 PCUOAM.......................................................................................................................46

5.3. TRANSMISSION NETWORK INTERFACES ....................................................................................46 5.3.1 PCM / optical interface ..................................................................................................46 5.3.2 Ethernet.........................................................................................................................47 5.3.3 X25 ................................................................................................................................47

6. O&M PROCEDURES ..................................................................................................................48 6.1. UPGRADES.............................................................................................................................48

6.1.1 OMC-R Upgrade ...........................................................................................................48 6.1.2 W-NMS Upgrade...........................................................................................................48 6.1.3 BSC, PCUSN, TCU & BTS upgrade .............................................................................49

6.2. INSTALLATION AND COMMISSIONING ........................................................................................50 6.3. MAINTENANCE ........................................................................................................................50 6.4. OPERATION ............................................................................................................................50 6.5. EXTENSION ............................................................................................................................50 6.6. TECHNICAL ASSISTANCE SUPPORT..........................................................................................51

7. FIELD INTRODUCTION ..............................................................................................................52 7.1. HARDWARE CONSTRAINTS.......................................................................................................52 7.2. SPARES CONSTRAINTS............................................................................................................52 7.3. BSS VERSION INTERWORKING ................................................................................................52 7.4. NSS INTERWORKING...............................................................................................................52 7.5. SGSN INTERWORKING............................................................................................................52 7.6. FUNCTION INTERWORKING.......................................................................................................52 7.7. ENGINEERING.........................................................................................................................52

7.7.1 metrics...........................................................................................................................52 7.7.2 rules...............................................................................................................................52

8. SUB SYSTEMS IMPACTS ..........................................................................................................53 8.1. OAM .....................................................................................................................................53

8.1.1 OMC-R ..........................................................................................................................53 8.1.2 W-NMS..........................................................................................................................53 8.1.3 PCUOAM.......................................................................................................................53 8.1.4 TOOLS ..........................................................................................................................53 8.1.5 NRP/upgrades BSS ......................................................................................................53

8.2. BSC 2G ..................................................................................................................................54 8.3. BSC E3..................................................................................................................................54 8.4. BTS ........................................................................................................................................55 8.5. TCU 2G ..................................................................................................................................55 8.6. TCU E3 ...................................................................................................................................55 8.7. PCUSN .................................................................................................................................55




8.7.1 PCUSN..........................................................................................................................55 8.7.2 PCUSN – PCUSPe2 .....................................................................................................55

8.8. TOOLS....................................................................................................................................56

9. ABBREVIATIONS AND DEFINITIONS.......................................................................................57 9.1. ABBREVIATIONS......................................................................................................................57 9.2. DEFINITIONS ...........................................................................................................................57




1. INTRODUCTION

1.1. OBJECT

The BSC upgrade offline protocol incurs a quite long interruption of service. This downtime must be improved to decrease as much as possible the interruption of service. This functional proposes numerous improvements to the offline and online upgrade and build on line protocols to decrease the BSC downtime.

Additionally, we also propose an enhancement to the offline TCU upgrade to decrease the TCU offline upgrade interruption of service.

1.2. SCOPE OF THIS DOCUMENT

The target BSC version for this feature is V16.

The feature is applicable to the following sub-systems:

Sub systems Y/N/I Example : L1M enhancement feature BSC 12000 N y for a sub-system which takes account the BSC e3 Y Feature n for a sub-system which does not take account PCUSN I i (independent) for a sub-system which is not PCUSN – PCUSPe2 I concerned by the feature TCU-TCB1 I TCU-TCB2 I TCU e3 Y S4000/S2000E-DCU2 I S4000/S2000E -DCU4 I S4000/S2000E-DCU2/DCU4 I S8000-BCF I S8000-CBCF I S2000L/H I e-cell I S12000 I S8002 I S8003 I S8006 I BTS 18000 I OMC-R Y PCU-OAM I CT2000 I ProPtima I W-NMS Y




1.3. AUDIENCE FOR THIS DOCUMENT

Draft: Nortel teams (R&D, PLM, Eng, Product Support…)

Preliminary: Nortel teams (R&D, PLM, Eng, Product Support…)

Standard: Nortel and customers teams.




2. RELATED DOCUMENTS

2.1. APPLICABLE DOCUMENTS

[A1] PE/BSC/DD/009734 Upgrade and Build on Line Performances Improvements on BSC & TCU e3

[A2] PE/SYS/DD/0425 Functional Note of BSC e3 Load Balancing

[A3] PE/BSC/DJD/9935 V15.0 PST BSC/TCUe3 Performances Results

[A4] PE/BSC/DD/010273 Software Functional Specification of the Newly Fault Tolerance

[A5] PE/BSC/DD/12715 SUPIN/TCU: Subsystem Functional Specification

[A6] DS/BSS/APP/0010 BSS Upgrade Procedures

[A7] UMT/RNC/DD/013942 Introduction of Background Flashing in BaseOS

2.2. REFERENCE DOCUMENTS

[R1] PE/BSC/DD/339 BSC E3 PRODUCT SPEC

[R2] PE/TCU/DD/0070 TCU E3 PRODUCT SPECIFICATION

[R3] PE/BSS/DD/3542 Upgrade Specifications for BSC/TCU e31

[R4] PE/BSS/DD/0052 Managed Object Dictionary volume 2

1 informations related to upgrade and build on line enhancements in this document obsolete/supercede information published in PE/BSS/DD/3542 V05.1/EN




3. BENEFITS

3.1. CUSTOMER/SERVICE PROVIDER BENEFITS

This feature aims at decreasing significantly the BSC and TCU downtime during a build on line or an upgrade offline. The target downtime to achieve for the BSC is less than 10 minutes whatever the offline upgrade type (type 6 or type 7) and even if transition module flash upgrade is required. The TCU offline upgrade downtime latency target is less than 10 minutes.

The main BSC improvements focus on BSC start-up latency following up an offline upgrade activation or MIB activation. The TCU improvement aims at ftp TCU loads on-line during an offline upgrade; hence without an interruption of service.

Additionally, this functional note describes BSC and TCU on-line upgrade enhancements that eases upgrade trace-ability and robustness.

The improvements are tracked by the following RFF:

Feature Description RFF # Type 4 Improvement

Build on Line Improvement

Type 6 & 7 Improvement

Release Phasing

Restart duration reduction-BSC3000

27689 N/A YES YES V16.0

TCU upgrade downtime reduction

28602 N/A N/A YES V16.0

benefit loss Cost ownership (day to day operation, fault management, upgrade, preventive/curative maintenance....)

X

Capacity Dimensioning (Connectivity, storage capacity…) Engineering (coverage, C/I…..) Traffic management (load sharing, compliance with recommendations …..) Robustness (restart/recovery enhancement…..) X

3.2. END-USER/SUBSCRIBER BENEFITS

Benefit benefit loss Voice quality (HO, call handling enhancement …)

Quality Of Service (call drop, availability..) New services




4. FUNCTIONAL DESCRIPTION

4.1. TCU OFFLINE UPGRADE

Currently, the TCU offline upgrade protocol requires to lock the TCU prior to activate the upgrade. This TCU lock incurs a very long interruption of service because the offline upgrade includes the TCU boards flash download via the LAPD channels; hence through a limited bandwidth. Recent performances measurements have shown that the TCU software download is longer than IN software download by an order of magnitude (see [A3] for details). A major improvement is to trigger an offline upgrade TCU without having to lock the TCU prior the offline upgrade activation; hence keep the TCU in service during the software download. This TCU offline enhancement is of premium importance since a single TCU handles 1500 Erlang capacity (half the traffic of a full BSC configuration).

Figure 1 shows the data flow of the new TCU offline upgrade protocol. This new offline protocol requires a new action TGE ‘backgroundTcuUpgrade’ defined on the 3GTranscoder admin object with a parameter that indicates if the upgrade is offline or online. This TGE backgroundTcuUpgrade will be used by standard upgrade NRP instead of the former TGE ‘softwareSet’. This TGE has an additional parameter to distinguish an offline from an online upgrade. Note that the former TGE softwareSet is maintained for backward compatibility purposes between an OMC V16 and BSCs in former release than V16.

BSC OMC TCU TGE backgroundTcuUpgrade (TCUe3, TC3vveeddpp.LIV,offline,…)

Upgrade offline req

2034:begin

ftp load to flash

Upgrade ack with reset

Init_dialog_req

Init_dialog_ack PCM configuration

Updates TCU S/W links to new version

Upgrade offline req 2034:begin

2024:cleared 2034:END

2024:cleared 2034:END

Upgrade ack w/o reset Startup Upgrade

TCU auto reset

Fuzzy Period

All Boards Flash Updated

SharedDisk

Check On upgrade

conditions OK

RGE OK




Figure 1: TCU Offline upgrade enhanced protocol

Figure 2 shows the new action TGE ‘restoreTcuReleaseConsistency’ when a upgrade TCU fails: If the OMC receives and RGE KO to backgroundTCuUpgrade, it must trigger a TGE restoreTcuSoftwareRelease whose purpose is to restore BDE/MIB consistency with respect to the TC3*.LIV label. In the figure, the corresponding actions at MMI are shown. In particular, the TGE ‘restoreTcuReleaseConsistency’ must be transparently attached to the backgroundReturnToPreviousVersion action defined at MMI.

Figure 2: Upgrade TCU failure

BSC OMC TCU

TGE backgroundTcuUpgrade (TCUe3, TC3vveeddpp.LIV,offline,…)

Upgrade offline req

2034:begin

Upgrade nack

Updates TCU S/W links to new versionMIB Updated with TC3*.LIV

TGE restoreTcuReleaseConsistency(TCue3 1, TC3*.LIV)

2024:major

2034:END

RGE OK

Restore BDE/MIB

Consistency

Check On upgrade

conditions OK

RGE KO

MIB Updated with TC3*.LIV

BackgroundActivateNewVersion

BackgroundReturnToPreviousVersion




4.2. BSC OFFLINE UPGRADE PRINCIPLE

The upgrade offline is applied when a new release breaks interface that prevents using an online upgrade. The offline upgrade principle is a three phases process:

• First phase consists in an upgrade flash of the interface node

• Second phase, in upgrading the control node

• Third phase consists in a reset the whole BSC to upload the new load and flash.

Currently, the first phase is done online. The second phase incurs a downtime that varies if control node flash upgrade is mandatory or not. The downtime latency ranges from 13 minutes to 20 minutes if control node flash update is necessary.

The main areas of improvements are two-folds:

• Control node flash update should be done online prior to the BSC reset

• BSC start-up duration should be decreased significantly

The first improvement consists to upgrade board flash without incurring any interruption of service. This online flash update is tricky since any premature board reset of an upgraded board raises an interoperability issue during an offline upgrade:. This may lead to situations where a board runs upgraded flash and load. The background flash protocol relies on hardware management to prevent such interoperability issues.

.




4.3. BSC START-UP

Currently, the BSC start-up has two main deficiencies: IN/TCU critical path is too long and the control node start-up serializes multiple steps:

• The passive OMU is always started up before TMUs whereas passive OMU is not part of the control node upgrade critical path. Any delay to start-up the passive OMU defers the TMU start-up; hence the recovery of service.

• Core processes start-up is serialized in two steps first active core processes and then passive. Each step serializes core process start-up by increasing order of importance. Any delay to start-up a core process defers the following core process to start up.

• IN critical path duration exceeds the SUP_IN start-up duration. This incurs an additional delay during BSC recovery since the control node has to wait for the end of IN critical path before being able to configure both IN and TCU.

Figure 3 describes the V15.1 BSC start-up chronogram during an offline upgrade activation. The duration of each step has been measured with a V15.0 load (see [A3] for details) and must be updated with accurate V15.1 performances measurements. The figure exhibits clearly the latency penalty incurred by the IN critical path duration which postpone IN configuration by 4min30sec and the TCU configuration by 3 minutes.

Figure 3: V15.1 BSC start-up chronogram during an offline upgrade activation

OMU Active

Config IN

Config TCU

2min30 V16.0

Total Downtime before first call : ~ 13min

IN Critical Path

7min TCU Clear Config

1min

Active Core Processes

3min30

OMU Passive

2min30

4min30

Passive Core Processes

3min30 3min

3min

Config IN

CN Ready to configure IN CN Ready to configure TCU

Config TCU

3min




Figure 4 below describes the new BSC start-up protocol with three major improvements:

•

• IN critical path duration decreases to two minutes (see section 4.7.1). This enables the IN critical path latency to overlap entirely with the active OMU start-up duration. Note that the requirement differs for IN and TCU critical path duration improvement. IN critical path must not exceed 2 minutes whereas TCU critical path can be a little bit longer without any impact on the overall BSC down time.

• .

Figure 4 shows the expected BSC start-up duration during an offline upgrade activation. The latencies provided in the figure below have been measured with a max dimensioning BSC configuration (54 cellgroups, 6 LSA on IN, 14 TMU1, max number of PCM) with a V15.1 load. All those latencies must be confirmed via new performances measurements with a V16.0 load. The important point is that the IN critical path latency reduction has a major impact on the BSC start-up overall latency. It is worth noting that the TCU configuration is launched much sooner than in Figure 3: TCU is configured as soon as IN fuzzy period is completed since the IN fuzzy period configure PCM Ater that enables LAPD communication with the TCU.




Figure 4: V16 BSC start-up chronogram after an offline upgrade

OMU Active Start-up

2min30

1mn45

Downtime before SS7 up: ~ 7min

IN CP

2min

TCU CC

1mn

Active/Passive Core Processes

3min40

0m30

5min30

SSN in service

0m30

OMU Passive Start-up

2min

TCU Conf

IN Config

3min30

Downtime before full duplex: ~ 9min30

0m30




4.4. BACKGROUND FLASHING OF TRANSITION MODULES

Currently, the control node flash updates are triggered offline during each board start-up. This incurs a high downtime penalty when flash updates are mandatory. This downtime penalty can be avoided if the flash update is triggered on-line prior to the control node reset.

Figure 5 exhibits the flash updates and their impacts on the BSC overall start-up latency. In the Figure 5 the flash actions appear in green and account for about 7 minutes penalty on the BSC startup latency compared with an offline upgrade without flash updates.

The background flashing raises numerous issues in case of premature board resets:

• Interoperability issue between load and flash: a board can recover with an upgraded flash and a former load. This interoperability issue requires ensuring backward interoperability between a load in N and flash in N+1. Note that this interoperability must be guaranteed for a gap of two releases.

• Interoperability between loads: offline upgrade incurs I/F break that prevent an upgraded board to interoperate with other not upgraded boards.

• Flash corruption: if a premature reset occurs before the completion of the flash upgrade. The premature board reset may lead to corrupt the flash that is upgraded. In that case the flash upgrade is postponed offline during the board recovery.

• Activity issue: the background flash must not preempt the CPU or any critical resources to let applications run during the flash upgrade. This request has strong impacts on the current flash protocol that attempt to flash large flash section with a very high priority thread.

Figure 5: BSC start-up duration with both OMU and TMU flash updates

OMU Active Flash

Config IN

Config TCU

2min30 1min

3min

3minTotal Downtime before first call : ~ 18min

IN Critical Path

7min TCU Clear Config

1min

Active CP

3min30

Total Downtime before full duplex : ~21min

TMU Flash Upgrade

8 min

OMU Active

2min30

Flash

1min

OMU Passive

2min30

1min

Passive CP

3min30




4.5. ACCURATE UPGRADE STATUS

Currently, the control node sends an upgrade status with only three different values:

• No-Upgrade

• Upgrade_in_progress

• Upgrade_performed

Those upgrade states are not convenient to follow up the upgrade progress from an operator point of view: the upgrade state is transiently set to “Upgrade_in_progress” during the board upgrade and updated to “Upgrade_Performed” immediately followed up of “No_upgrade” state as soon the board has been upgraded. Moreover, there is no upgrade status related to the flash upgrade.

Figure 6 shows the new global upgrade status that applies to IN, TCU and CN to help the operator to follow up both online and offline upgrade for all upgrade types 3/6/7/4:

• NO_UPG this is the default equipment upgrade status meaning either that no upgrade is on-going on the equipment or the upgrade has just completed successfully or not.

• ‘UPG_IN_PROGRESS’ is set if an upgrade activation has been received and the pre conditions hold. If the pre conditions check fails then the upgrade status remains to NO_UPG and a RGE KO is sent to the OMC. The equipment remains in state UPG_IN_PROGRESS until either the last board is upgraded successfully, the upgrade is frozen or the upgrade failed.

• UPG_FAILED this state is set if on-going upgrade has failed. On the control node, an on-going upgrade fails if an automatic fallback is triggered (see [R3] for accurate fallback conditions). Note that if an upgrade activation is rejected during pre conditions check the equipment remains in the state NO_UPG. In other words, an UPG_FAILED state can be set if the equipment state is either UPG_IN_PROGRESS or UPG_FROZEN

• UPG_FROZEN is set as soon as the control node upgrade manager freeze the upgrade. Note that the UPG_FROZEN state applies only to the control node and has no equivalent on IN/TCU. The upgrade status remains ‘UPG_FROZEN’ until the upgrade is resumed or canceled.

• ‘UPG_PERFORMED’ is set as soon as all boards that needed to be upgraded have been upgraded successfully. This is transient upgrade state: in case of an offline upgrade the global upgrade state is set to NO_UPG after the equipment reset (see Figure 11 & Figure 12). In case of an online upgrade the upgrade status is set to NO_UPG’ as soon as the alarm 2034 upgrade completed has been sent.




Figure 6: Global equipment upgrade states change

NO_UPG

UPG_IN_PROGRESS

UPG_FROZEN

UPG_PERFORMED UPG_FAILED

Upgrade Activation

Resume Upgrade Suspend Upgrade

Cancel Upgrade

Pre Conditions Failed RGE KO sent to OMC

Automatic Transition

Upgrade Completed

Restart In Previous Version

Upgrade Failed




The board upgrade states are the following as shown in Figure 7:

• NO_UPG: the board is in the right version and the global upgrade is completed.

• UPG_NEEDED: the board has been checked and is not in the new version, and needs to be upgraded

• UPG_IN_PROGRESS: the board is currently being upgraded

• UPG_FLASH_UPDATED: the board flash has been upgraded, but board has not yet been reset to put load in RAM. Note that this upgrade state differs between CN and IN. On the CN, this state is only updated in case of background flashing; not during offline flashing. On the IN side, this state is set both during offline or online upgrade since flash updates is always done in background.

• UPG_PERFORMED: the board has been upgraded during an online upgrade and probation period has started but the global upgrade is not yet completed. Note that this state is not relevant with offline upgrade since all boards are upgraded in a single shot via the equipment reset.

• UPG_FAILED: the board upgrade failed

Figure 7: Board Upgrade Status

NO_UPG

UPG_IN_PROGRESS

UPG_FAILED

UPG_PERFORMED

UPG_FLASH_UPDATED

Load Updated-no flash updated

Flash Updated

Background Flash Upgrade Timeout

UPG_NEEDED

Board Upgrade Request

Load Updated

Board Recovery Fails

C

Boards is ENABLE/ONLINE

needs to be Upgraded

Board upgraded offline

Init Global Upgrade Frozen

Global Upgrade Frozen




Board Recovery Successful

Online only state Online and offline upgrade state




There are dependencies between the global upgrade states and the board upgrade states:

• UPG_NEEDED or UPG_PERFORMED must be sent for each board right after the pre conditions hold and the global upgrade status is set to UPG_IN_PROGRESS. The UPG_PERFORMED state at this stage reflects the boards that are not impacted by the upgrade. In other words, a board upgrade cannot be set to UPG_NEEDED or UPG_PERFORMED if the global upgrade state is not UPG_IN_PROGRESS.

• The board upgrade state is updated to NO_UPG as soon as the global upgrade state is set UPG_PERFORMED.

Figure 8 shows the correlation between the global upgrade state change and the board upgrade changes during an online upgrade. The figure assumes that the global equipment state is set to NO_UPG when it receives an upgrade activation request. First the equipment check whether the pre conditions hold. If the pre conditions hold then the global state of the equipment is updated to UPG_IN_PROGRESS meaning that the upgrade activation will proceed. In the meantime, the equipment sends an upgrade state for each board. The board upgrade state is set to UPG_NEEDED for any ENABLE/ONLINE boards2. Note that SUP_IN/TCU/CN is responsible for setting the ENABLE/ONLINE board state to UPG_NEEDED.

Once the latest board has been upgraded successfully the global upgrade status is updated to UPG_PERFORMED and boards upgrade status is set to NO_UPG meaning that the upgrade is completed. The global upgrade state remains in UPG_PERFORMED until the equipment is reset and updated to NO_UPG when the equipment recovers. Note that it is the responsibility of SUPM_IN/TCU/CN to set the board status to NO_UPG for all boards.

The state change are emitted by the equipment but mediated by the corresponding supervision software elements. Those state changes must be clearly reflected on GUI interface of the OMC.

2 An alternative would be to set only impacted boards to UPG_NEEDED whereas non impacted boards would remain to NO_UPG. For sake of simplicity, all ENABLE/ONLINE boards are set to UPG_NEEDED in a first approach.




Figure 8: Dependencies between board and global upgrade states

Pre Conditions Check

UPG_IN_PROGRESS

UPG_NEEDED_FOR ALL BOARDS ENABLE/ONLINE E

OMC BSC/TCU SUP_IN/TCU/CN

Upgrade last board

UPG_PERFORMED

NO_UPG

NO UPG

UPG_EQP_IND(UpgInProgress)

UPG_MODULE_IND(UpgPerformed)

UPG_EQP_IND(|UpgPerformed)

TGE Upgrade Activation

RGE OK

2034: Upgrade completed

2024: Upgrade completed

Upgrade board

Upgrade flash

UPG_MODULE_IND(UpgFlash)

If flash update needed

UPG_FLASH_UPDATED

UPG_PERFORMED

NO_UPG _FOR EACH BOARD

UPG_EQP_IND(|NoUpg)

UPG_FLASH_UPDATED

Mis en forme : Police :14 pt



Mis en forme : Police :12 pt,Gras, Police de script complexe:12 pt, Gras




4.6. UPGRADE ONLINE ENHANCEMENTS

The upgrade online enhancements are numerous:

• Improve control node upgrade checks to guarantee that no service interruption can occur during an on line upgrade.

• Decrease on-line upgrade of OMU by leveraging OMU application restart in the context of an on-line BSC upgrade.

4.6.1 TMU UPGRADE ENHANCED PROTOCOL

Currently, the upgrade control node assumes that the control node can sustain at most one TMU reset without impacting the call processing service. In consequence, the upgrade control node manager freezes the TMU upgrade in case of an unexpected TMU reset during TMUs upgrade. The upgrade is eventually resumed when the failed TMU recovers in case of software error or is replaced in case of hardware error. Note that the TMU recovery does not guarantee that core processes are duplex again.

Actually, the number of TMU resets that can sustain the control node varies a lot from one system to another: a lightly loaded control with numerous extra (spares) TMUs may sustain multiple concurrent TMUs resets without any impact on the call processing service whereas a heavy loaded control node sustain only one TMU reset at a time.

The upgrade control node manager must check that core processes are still duplex before resetting another TMU instead of checking that no TMU has failed.

If the upgrade control node manager has to freeze TMUs upgrade because some core processes are no more duplex then the 2035 alarm notification should precise if this is a permanent freeze state (see section 4.6.2). Here is the new list of 2035 alarm probable cause that will be sent by the upgrade control node manager:

• Cause 0 : Board failure

• Cause 1 : OMU Failure

• Cause 2 : CC1 Failure

• Cause 3 : TMU applications not duplex

• Cause 4 : ATM_RM are not hot duplex

• Cause 5 : CC1 link failure

• Cause 6: TMU Failure

• Cause 7: Permanent Freeze State on TMU




This core process duplex check impact also the pre-conditions check that are verified at the beginning of the upgrade control node. The control node upgrade manager must check that the all core processes hosted on TMU are duplex before triggering the upgrade activation. If the core processes are not duplex during pre-conditions check then the upgrade control manager rejects the online upgrade request and sends back to the OMC an RGE KO with an accurate error cause. It is up to the operator to trigger manuals operations to restore core process duplexity. For instance, the operator may trigger an on-line load balancing reconfiguration that will eventually restore core process duplexity (see [A2] for details on load balancing reconfiguration).

Figure 9 shows the state chart of the enhanced TMU online upgrade protocol. Note that the new protocol requires the upgrade control node manager to subscribe to the load balancing distribution subject that provides core processes distribution state after each load balancing event (TMU insertion, TMU reset, load reconfiguration,…). The upgrade control node manager remains in the freeze state until a new LB event is received. Conversely, the upgrade control node check the GSM or SS7 duplex properties each time it receives a new event from load balancing. It is worth noting that an insert TMU event does not necessarily lead to recover core processes duplexity (see section 4.6.2).

Note that the upgrade control node manager still upgrades a TMU at a time; hence must wait for the upgraded TMU to recover before upgrading the following one.

Figure 9: TMU on-line upgrade state chart

Upgrade first GSM TMU

Upgrade IN

upgrade next GSM TMU

Wait DMS 5 min Probation Timer

Upgrade active SS7 TMU

Freeze Upgrade

Freeze Upgrade

TMU upgraded successfully

yes

first TMU upgraded successfully

yes

all GSM TMU upgraded Probation timer expiration

yes

LB new event no

All GSM CP duplex ?

All GSM CP duplex ?

no LB new event

Freeze Upgrade

no LB new event

resumeUpgrade

Resume Upgrade

All GSM CP duplex ?

NOP

Resume Upgrade




The active SS7 TMU must be upgraded last in order to avoid multiple SS7 linkset drops which may be understood by DMS like link issues and consequently probation periods activation. See 4.6.4.

In case of online upgrade, the upgrade of the first TMU is a significative step: after probation time, IN upgrade is launched and the CN upgrade continues.

So, to start the upgrade of TMUs set with a GSM CP TMU is more representative for the online upgrade probation time. So, the following simple rule is defined: passive SS7 CP TMU and then Active SS7 CP TMU.

4.6.2 PERMANENT FREEZE STATE

The core processes duplexity check may stick the TMU on-line upgrade in a freeze state although the failed or reset TMU has successfully recovered. This permanent freeze state may arise in some rare circumstances where load balancing does not find enough room among live TMUs (including the recovered one) to fit the missing active or passive core processes due to the degradation of the initial distribution.

Although this permanent freeze state is assumed to be rare enough given engineering rules it cannot be ignored. The operator has the ability to force the upgrade with a limited service impact.

The operator is notified of a permanent freeze state through a 2035 alarm with a new probable cause 7: “Permanent Freeze State on TMU”. From an upgrade control node manager, a permanent freeze state is a freeze state where some core processes are simplex or nullex whereas the number of TMUs up and running is equal or greater than the number of TMUs at the beginning of the TMUs upgrade.

Once the operator has triggered a ‘resumeUpgrade’ TGE (see 4.6.3) to continue the on-line TMU upgrade, the upgrade manager will complete the upgrade even if some core processes are missing. In other words, the alarm 2035 with probable cause set to “Permanent freeze state on TMU” is only sent once by the BSC.

4.6.3 RESUMEUPGRADE TGE

There is a “resumeUpgrade TGE defined for CN object that enables the operator either to continue on the upgrade or to trigger a “fallback”.

When the upgrade is frozen, a notification (ano 2035 on the CN object with the appropriate cause) is sent by the BSC to the OMC-R.

The upgrade is frozen in the following cases:

• During CC1 upgrade see [A1] paragraph 4.2 and sub-sections

• After the successful upgrade completion of the first TMU, in case of module permanent error like hardware failure and as seen in 4.6.2, in case of TMU Core process distribution failure.




The Operator at the OMCR is warned that some service disruption may result in use of the resume TGE. No automatic recommendation is made to the operator on whether to fallback or upgrade; the operator must assess the upgrade progress/situation and make a decision. Once the operator has elected to resume an upgrade or fallback, all future freeze conditions on the CC1 or the TMU are ignored.

Given that the resumeUpgrade TGE with action fallback can be handled partway through the TMU upgrade, a change IN interface node upgrade behavior is required in order to downgrade IN. The IN downgrade is not started until the OMU SWACT has been completed, since the IN software version is pointed to by the /OMU directory on the active OMU. The upgrade completed may be sent upon receiving the response from IN.

If an IN upgrade failure occurs in the same CN conditions i.e. fallback handled partway through TMU upgrade, the behavior is similar in case of IN upgrade failure: as there is no automatic fallback, IN stays in this state until the OMU SWACT has been completed

4.6.4 REDUCED TMU SS7 PROBATION TIMER

Currently, the upgrade control node upgrades last the active TMUs SS7. A probation timer of 11 minutes is set between the upgrade of the preceding TMU and the upgrade of the active TMU SS7. This probation timer is required to avoid a 5 minutes penalty timer set on DMS Nortel if a SS7 link breaks twice during an 11 minutes probation timer.

Both the penalty timer and the probation timer have been decreased to respectively 1 and 4 minutes with NSS 17 delivery and are available through patches in former NSS releases. This shorter penalty and probation timer values must be reused on the control node to shorten a little bit the TMU online upgrade. This means that the former 11 minutes timer before upgrading the active TMU SS7 must be set in V16.0 to 5 minutes.




4.6.5 Improved OMU upgrade online

The online OMU upgrade could leverage from OMU application restart to roughly halve the latency to upgrade both OMUs. The Figure below exhibits a comparison of V15.1 and V16.0 chronogram of the online OMU upgrade. Currently, the OMUs upgrade incurs three OMU reset; each OMU reset accounting for 5 minutes latency. The proposed improvement is to replace two of three OMU resets by OMU restarts. Precisely, in V16 passive OMU upgrade triggers an OMU restart whereas the OMU switch over (i.e swact) still require an OMU reset to transfer the OMU activity to passive mate.

It is worth noting that OMU restart bypass the AIX bootstrap; hence preventing any upgrade of AIX. However, this limitation is not expected to be cumbersome since AIX upgrade is scarce and may be applied via an offline upgrade. The improved OMU online protocol accommodates with OMU_TM background flashing since OMU_TM flash is uploaded via an OMU restart provided that the new flash does not introduce any I/F break.

Figure 10: OMU online upgrade chronograms

Upgrade Passive OMUB

& Reset Reset

Active OMUA

Upgrade Passive OMUA

& reset

& Restart

Reset Active OMUA

5min30

2min30

30sec

18min

12min

30sec 30sec 30sec

V16


& Reset Reset

Active OMUA

Upgrade Passive OMUA

& reset



& Restart

Reset Active OMUA

18min

30sec 30sec 30sec

V16

30sec 30sec 5min30

5min30 2min30

5min30

Passive OMUBUpgrade

Passive OMUA& Restart

V15.1




4.7. SPECIFICATION

This section details sub system impacts of each feature.

4.7.1 IN/TCU CRITICAL PATH IMPROVEMENTS

In V16, the IN/TCU critical path behavior will be improved. The objectives of this improvement are twofold:

• Decrease significantly the amount of time taken to complete the critical path

• Make the critical path behavior more deterministic and robust

A number of changes have been planned to achieve these objectives.

Performances Improvements The latency improvement per step on the nominal critical path latency is detailed here below:

• Activity determination: 40 seconds

• Elimination of “get equipment list” step: 64 seconds

• Removal of add child serialization: 60 seconds

• RM recovery speedup: 146 seconds for fully loaded IN, 150 seconds for fully loaded based on 130 secs for IEM OOS test and 1X16 or 20 RMs

Those savings account for 310 sec of the overall nominal critical path latency. The expectation is to shorten the critical path latency to 2 minutes (*) in V16 instead of 7 minutes in V15.

(*) an extra duration of 1 – 1.5mn is added in case of power off/on because of tests on all cards

4.7.2 IN/TCU OFFLINE UPGRADE ROBUSTNESS

The current offline upgrade implementation cannot be relied upon to deliver broken CEM-RM interfaces. Consequently, a new IN/TCU offline upgrade strategy will be implemented in V16 which will enable the reliable delivery of loads containing broken interfaces. Without these upgrade enhancements, if the IN or TCU experienced a premature reset before all cards had been upgraded, the node would only be recoverable via the TML interface. In the case of a remote IN or TCU, this could result in a lengthy outage.




The approach described here relies on the fact that broken CEM-RM interfaces only become an issue if they prevent communication with the CN. Broken interfaces can be activated if there is a premature global or card reset caused by some fault. Therefore, this strategy ensures that no matter when during the upgrade process a premature reset occurs, there will always be a way to align the software versions of the active CEM and active interface card. During an offline upgrade, if secondary flash loading fails for both CEMs, Upgrade Manager will still proceed with direct transfer of interface load to the interface cards. In this case, if a premature node reset occurs, there is a possiblility of interface break.

This new offline upgrade strategy comprises two new behaviors:

1. the order in which the cards are upgraded is changed: parallel loading of RM’s starts only after both the CEM’s are upgraded. The simultaneous loading of the RM’s occurs only for the case of RM mate loading or Secondary flash to RM loading

2. the new load for the interface card will be stored in the CEM flashes prior to beginning the upgrade

A CEM flash partition will be used to store the interface card’s load until the upgrade has completed successfully. It will be referred to here as the CEM’s “secondary flash”. The CEM flash partition reserved for the CEM’s software image will be referred to as its “primary flash”.

The flow of events during an offline upgrade of a node from version ‘N’ to version ‘N+1’ is as follows.

The term “interface card” (I/F) or “interface rm” is used to refer to the card involved in the Critical Path that is the ATM RM in the IN and the IEM in the TCU. The term “Protection group” is used to refer to the other cards SRT and IEM in IN and TRM in TCU.

Initially, the flash of all RMs and the CEMs contain the previous load: (Please note that all boxes in the following figure represent the flash of the cards mentioned.)

1 - The secondary flash of the Passive CEM is loaded with the new interface load through FTP loading:

N N

Interface RMs

Sec. flash

Active CEM

Passive CEM

N N flash

N N

N N N N

A P A

A A

P

P P

Protection Group 1

N N

Protection Group 3 Protection Group 2




2. The secondary flash of the Active CEM is loaded with the new interface load using mate loading from the secondary flash of the Passive CEM:

3. The flash of the passive CEM is loaded with the CEM load using FTP:

N N

Interface RMs

Sec. flash

Active CEM

Passive CEM

N N+1 flash

N N

N N N N

A P A

A A

P

P P

Protection Group 1

N+1 N+1


N N

Interface RMs

Sec. flash

Active CEM

Passive CEM

N N flash

N N

N N N N

A P A

A A

P

P P

Protection Group 1

N+1 N+1


N N

Interface RMs

Sec. flash

Active CEM

Passive CEM

N N flash

N N

N N N N

A P A

A A

P

P P

Protection Group 1

N N+1





4. The flash of the Active CEM is loaded with the new CEM load using mate loading from the flash of the Passive CEM:

5. Once the flash of both CEMs are upgraded, simultaneous loading of RMs begins. At any given time, there can be a maximum of 1 FTP loading, and 4 mate loading in progress for RMs.

First, the FTP loading for the Passive RM in Prot Group 1 is initiated. Simultaneously, the mate loading of the flash of the Passive interface RM from the secondary flash of the active CEM begins:

6. Once the FTP loading of the flash of Passive RM in Prot Group1 is completed, the FTP loading of Passive RM in Prot Group 2 begins.

Meanwhile, the mate loading of the Active interface RM from the flash of the Passive interface RM begins.

Similarly, the mate loading of the Active RM in Prot Group 1 from the flash of the Passive RM in the same Prot Group begins.

N+1 N+1

Interface RMs

Sec. flash

Active CEM

Passive CEM

N+1 N+1 flash

N+1 N+1

N N+1 N+1 N+1

A P A

A A

P

P P

Protection Group 1

N+1 N+1


N N

Interface RMs

Sec. flash

Active CEM

Passive CEM

N+1 N+1 flash

N N

N N N N

A P A

A A

P

P P

Protection Group 1

N+1 N+1





7. Once the FTP loading of the flash of Passive RM in Prot Group2 is completed, the FTP loading of Passive RM in Prot Group 3 begins.

Meanwhile, the mate loading of the Active RM in Prot Group 2 from the flash of the Passive RM in the same Prot Group begins.

8. Once the FTP loading of the flash of Passive RM in Prot Group3 is completed, the mate loading of the Active RM in Prot Group 3 from the flash of the Passive RM in the same Prot Group is done.

N+1 N+1

Interface RMs

Sec. flash

Active CEM

Passive CEM

N+1 N+1 flash

N+1 N+1

N N+1 N+1 N+1

A P A

A A

P

P P

Protection Group 1

N+1 N+1

4min Protection Group 3 Protection Group 2

N+1 N+1

Interface RMs

Sec. flash

Active CEM

Passive CEM

N+1 N+1 flash

N+1 N+1

N N N N+1

A P A

A A

P

P P

Protection Group 1

N+1 N+1





9. Once all the flash loads are upgraded, an Upgrade Complete message is sent to the Control Node. When the Platform Reset signal is sent from the Control Node, all cards are reset and the RAM of all cards are updated with the ‘N + 1’ load from the flash.

If a premature reset was to occur at any time during the upgrade, the system will always be in a position to align the active CEM and active I/F card with the new load so that it can regain connectivity with the CN. The key to achieving this is to focus on the two CEMs and the two interface cards. The phase between upgrading the first CEM and upgrading the last interface card will be called the “interface alignment phase”. During this phase, if a premature reset occurs, the strategy will be to complete the interface alignment phase as soon as possible and request a reset to activate the new loads. This ensures connectivity to the CN. If a single card which has been upgraded gets reset (i.e. not a global reset) that card will be locked until the upgrade has completed. Note that this strategy works equally well for downgrades.

Precondition step 1 is enforced to avoid the situation where a CEM might be out of service at the start of the upgrade but after a premature reset become active without having an I/F card load available. If the mate CEM is simply out of service but its flash can be upgraded, the upgrade will proceed.

N+1 N+1

Interface RMs

Sec. flash

Active CEM

Passive CEM

N+1 N+1 flash

N+1 N+1

N+1 N+1 N+1 N+1

A P A

A A

P

P P

Protection Group 1

N+1 N+1





4.7.3 BACKGROUND FLASHING

The background flash upgrade is triggered by the upgrade control node manager when flash update is required prior to the load upgrade. The flash upgrade is triggered on a board basis both during an offline or an online upgrade. However, the online upgrade triggers a board flash upgrade at a time although an offline upgrade will trigger concurrent board flash. The number of concurrent flash updates should be throttled to a configurable threshold stored in a configuration file out of the MIB.

The background flash protocol applies only for XX_TM boards on the control node that needs to be flashed. In a first approach, the background flashing is limited to TMU1_TM and OMU1_TM boards. This is due to the fact that flash on TM includes baseOS; hence are modified on a frequent basis; which is not the case of other boards. In particular TMU2 flash contents are assumed to be very light; hence are assumed to change rarely during TMU2 lifecycle.

The background flash is done in best effort without any repetition. If a background flash upgrade fails it will be triggered offline after control node reset. This requires that HM support both offline and online flash update protocols. For that purpose, the offline flash upgrade will be still delivered within transition module flash whereas the online flash upgrade protocol will be delivered within transition module loads. The online flash upgrade protocol will supersede the offline one during board initialization.

The interoperability issue between N and N+1 during an offline upgrade is solved by preventing a board that reset to start-up during an offline upgrade. HM is responsible to suspend the start-up of a board whose flash and load have been already upgraded. HM state must resist to an OMU swact; hence board state must be notified to HM passive mate. Note that this requirement applies only to an offline upgrade, not to an online upgrade since N and N+1 load are assumed to interoperate during an online upgrade.

The background flashing should be throttled on a granule basis corresponding to a flash sector. Note that this does not prevent to fetch the whole flash on the target TM board provided that there is enough free memory.

The background flashing requires to modify the flash driver that mask interruptions during erase phase.




Figure 11 shows the TMU upgrade online data flow with background flashing. At the beginning, all upgrade status is set to ‘UPG_NEEDED’.The upgrade control node manager iterates on each TMU to upgrade one after another. The upgrade control node manager sends a blind flash update request to HM to trigger a background flashing for that module. Hardware management sends back a reply to the upgrade control manager indicating If the flash has been updated or not. If the flash has been updated, the upgrade control node manager notifies the supervision control node that the upgrade status must be set to ‘UPG_FLASH_UPDATED’. Once upgrade control manager has received back the reply from HM, it reset the module and updates upgrade status via notifications send to SupCn. The upgrade status is updated to ‘NO_UPG’ once all TMUs have been upgraded.

Figure 12 shows the upgrade offline data flow with background flashing. The main changes with current offline upgrade protocol are the following:

• The upgrade control node manager is responsible for requesting flash upgrade to HM.

• The upgrade control node manager notifies the supervision control node of board upgrade status change.

The upgrade control node manager governs the degree of concurrency of modules flash based on a configurable threshold. In the Figure 12, the upgrade control node manager iterates on sending flash module requests to hardware management up to some threshold. The upgrade control node manager notifies the supervision control node of any upgrade state change.

The board upgrade status pass to ‘UPG_PERFORMED’ as soon as the software links on the shared disk have been updated but before control node restart. The flash and load software links are updated at the same time which may raise an interoperability issues in case of premature board reset.

Note that some boards upgrade states differ depending on the flash updates: some boards iterates on three states NO_UPG, UPG_NEEDED, UPG_PERFORMED if no flash update while other boards upgrade states iterate through NO_UPG, UPG_NEEDED, UPG_IN_PROGRESS, UPG_FLASH_UPDATED, UPG_PERFORMED,




Figure 11: TMU online upgrade data flow

UpgradeCn HM SupCn

UPG_IN_PROGRESS

UPG_PERFORMED

UPG_NEEDED FOR ALL BOARDS ENABLE/ONLINE

Only if flash has changed

UPG_IN_PROGRESS

Disable Board Freeze

UPGRADE_MODULE_IND(UpgInProgress) Update Flash & Load S/W links

Module Reset

Upgrade one Module Flash

UPGRADE_MODULE_IND(UpgFlashed)

UPGRADE_MODULE_IND(UpgPerfomed)

FLASH_UPGRADE_MODULE_RSQ

FLASH_UPGRADE_MODULE_RSP

NE

XT

TM

UT

OU

PGR

RA

DE

UPG_FLASH_UPDATED

Module Ready

Check CP Duplex

Upgrade Empty Slot

Wait Restart

UPGRADE_BSC_IND(StartOfUpgrade) NO_UPG

UPGRADE_BSC_IND(EndOfUpgrade)

NO_UPG FOR EACH BOARD

Pre Conditions hold

UPG_PERFORMED UPG NEEDED Board Upgrade Status UPG PERFORMED State applies only to li d

Mis en forme : Police :7 pt,Police de script complexe :9 pt





Figure 12: TMU upgrade offline data flow

Disable Board Freeze

Update Flash and Load S/W links

Update Load & Flash S/W links empty slots

Upgrade one Module Flash

UPGRADE_MODULE_IND(UpgInProgress)

UPGRADE_BSC_IND(EndoFUpgrade)

FLASH_UPGRADE_MODULE_RSQ

FLASH_UPGRADE_MODULE_RSP

UpgradeCn HM SupCn

UP

TO

TH

RE

SHO

LD

UPG_NEEDED_FOR ALL BOARDS ENABLED/ONLINE

UPG_IN_PROGRESS

UPG_FLASH_UPDATED

UPG_PERFORMED

UPG NEEDED Board Upgrade Status UPG PERFORMED Global Upgrade Status

UPG_IN_PROGRESS

NO_UPG FOR ALL BOARDS ENABLE/ONLINE

Only if flash has changed

NO_UPG

UPGRADE_BSC_IND(StartOfUpgrade)

OMU active reset

NO_UPG

UPGRADE_MODULE_IND(UpgFlashed)






4.8. FEATURE INTERWORKING

N/A

4.9. PERFORMANCES

It is important to keep in mind that the drvivers of the modifications here described are the following ones:

• To reduce the service interruption duration for the Build on Line and the upgrade type #6/#7, The ideal target is 5min (RFF 27689)

• To reduce the duration of each one in accordance with the maintenance window constraints.

For configuration conditions: 14TMUs, 54CG, 500sites, 1000TRX, 126E1/168T1.

Regarding TCU offline upgrade downtime, it is expected to decrease to 7min, critical path and fuzzy period duration included.

Regarding BSC offline upgrade downtime, it is expected to decrease significantly in case of transition module flash updates by leveraging background flashing (around 7min)

Regarding Build on Line downtime, the duration is expected to be 1min shorter than in V15.1 (several improvements were already done in the previous release see [A1]. The improvements here come from the critical path and fuzzy period enhancement of both IN and TCU).

Regarding BSC start-up power off/on duration, it is expected to be 3min shorter than in V15.1 (thanks IN/TCU enhancements mainly).

The Table 1 shows some duration estimations with configuration conditions: 14TMU1s, 54CG, 500sites, 1000TRX, 126E1.

Table 1: duration expected for V16

downtime duration

Build on Line ~6min -

Upgrade Type#6 ~6min -

TCU offline upgrade ~7min30 -

BSC start-up power off/on NA ~9min30




5. INTERFACES

5.1. GSM INTERFACES

5.1.1 AIR INTERFACE

No impact

5.1.2 A INTERFACE

No impact

5.1.3 LB INTERFACE

No impact

5.1.4 ABIS INTERFACE

No impact

5.1.5 ATER INTERFACE

No impact

5.1.6 AGPRS INTERFACE

No impact

5.1.7 Gb INTERFACE

No impact




5.2. O&M INTERFACES

The syntax of the following parameters may be changed until the CuR.

5.2.1 LOCAL MANAGER (MAN MACHINE INTERFACE)

5.2.1.1 DATA MODEL

The new upgrade status must be also reflected in getDD on CN/IN/TCU. This has impact on the data model.

The MMI commands related to the new TGE backgroundTcuUpgrade and restoreTcuReleaseConsistency must appear on the software object of the TCUe3 although the TGE are defined on the G3Transcoder object of the MOD.

The background TCU upgrade will be exported at MMI with the following commands:

• ‘ActivateNewVersion’ • ‘ReturnToPreviousVersion’.

The ‘ActivateNewVersion’ and ‘ReturnToPreviousVersion’ commands must trigger:

• The current softwareSet TGE in front of a BSC e3 V15, • The new backgroundTcuUpgrade TGE in front of a BSC e3 V16.

The RestoreTcuReleaseConsistency must be triggered transparently at MMI and be mediated to the command ‘ActivateNewVersion’ or ‘ReturnToPreviousVersion’.

Even if there is a new TGE on MOD, the same mechanism as today should be used for operators. So in case of a normal upgrade/fallback MMI command usage, the backgroundTcuUpgrade TGE must be sent in front of a BSC e3 V16.

In case of MIB update after error on normal upgrade (resp. fallback) MMI command, the operator should select the opposite MMI command of fallback (resp. upgrade) and CM should send the restoreTcuReleaseConsistency TGE in front of a BSC e3 V16.

If ActivateNewVersion (resp. ReturnToPreviousVersion) MMI command led to RGE status OK, then fallback ReturnToPreviousVersion (resp. ActivateNewVersion) is performed via backgroundTcuUpgrade TGE.

If ActivateNewVersion (resp. ReturnToPreviousVersion) MMI command led to RGE status NOK, then fallback ReturnToPreviousVersion (resp. ActivateNewVersion) is performed via restoreTcuReleaseConsistency TGE.

5.2.1.2 ALARMS & NOTIFICATIONS

No new alarms are required for this feature.




5.2.1.3 EVENT

The upgrade status values are changed to take into account new values. Those upgrade state changes must be reflected on MMI to help the operator follow up an-going upgrade.Note that the definition is backward compliant with older OMC/BSC releases and consistent among all managed objects including BSC 2G that uses DUPLICATE_IN_PROGRESS for background downloading and UPG_ABORTED.

Message definition This notification is sent to the operator each time the upgrade status of one of the managed object is modified.

Message operation code State Change Managed Object CN/IN/3Gtranscoder

CEM/IEM/ATM_RM/8KSRT/TRM/OMU/TMU/CC1 Event type Notification of upgrade status Structure definition

BSC 12000 BSC e3 BSC Product No Yes

FIELD MEANING Upgrade status 0: UPG_NOT_INPROGRESS

1: UPG_IN_PROGRESS 2: UPG_PERFORMED 3: UPG_NEEDED 4: UPG_ABORTED 5: DUPLICATE_IN_PROGRESS 6: UPG_FROZEN 7: UPG_FAILED 8: UPG_FLASH_UPDATED




5.2.1.4 ACTIONS & INTERROGATIONS TGE & RGE

The new upgrade offline TCU protocol requires a TGE upgrade activation defined on the G3Transcoder admin object. The upgrade activation TGE will be used for both offline and online TCU upgrade activation. A new parameter UpgradeType is defined to distinguish an offline from an online upgrade. The OMC-R must not lock the TCU prior to an upgrade offline in case of backgroundTcuUpgrade.

Message definition TGE sent to activate upgrade on TCU e3 Message operation code backgroundTcuUpgrade Managed Object G3Transcoder Event type Structure definition


Description Type Size Remark FileSetRef M STRING Contains .LIV info UpgradeType M INT 0: online upgrade

1: offline upgrade




The former TGE softwareSet with reducedVersion set to 3 is replaced by an explicit action TGE ‘restoreTcuReleaseConsistency’ defined on the G3Transcoder admin object. This TGE is triggered by OMC to restore BDE/MIB consistency with respect to TCU software release TC3*.LIV label stored both in BDE and MIB. Note that this TGE needs a single FileSetRef parameter, the UpgradeType parameter is not needed for that TGE.

Message definition TGE sent to return to restore BDE/BDA consistency Message operation code restoreTcuReleaseConsistency Managed Object G3Transcoder Event type Structure definition


Description Type Size Remark FileSetRef M STRING Contains .LIV info




The resume upgrade TGE defined on the control node managed object provides the operator the ability to resume or fallback the on-line upgrade when in a freeze state. This TGE is already defined in managed object dictionary volume 2 (see [R4]). Note that the current TGE definition includes an additional parameter ‘resumeImpact’ which is useless and must be removed from MOD V1 In case of fallback, the OMC must roll back the BDE software version to restore consistency with the BSC/MIB when it receives a RGE OK from BSC.

The resumeUpgrade TGE is only relevant to online upgrade. This TGE must be disabled for all other BSC offline upgrade flavors (type 3/6/7). OMC must trigger the same check than during a standard upgrade activation.

The upgrade control node manager will send the standard couples of 2035 alarm when it proceeds the ‘resumeUpgrade’ TGE whatever the resume type selected.

The resumeType field must be shown at MMI to be selected by the operator.

Message definition TGE sent to resume upgrade on control node Message operation code resumeUpgrade Managed Object CN Event type Structure definition


Description Type Size Remark ResumeType M INT 0: upgrade

1: fallback

5.2.1.5 BSC COUNTERS

No Impact

5.2.2 EXTERNAL MANAGER (Q3 / 3GPP INTERFACE)

5.2.2.1 DATA MODEL

See MMI interface

5.2.2.2 ALARMS & NOTIFICATIONS

See MMI interface

5.2.2.3 COUNTERS

See MMI interface




5.2.2.4 MISCELLANEOUS

No impact

5.2.3 SDO

No impact.

5.2.4 PCUOAM

No Impact

5.2.4.1 DATA MODEL

No impact

5.2.4.2 ALARM & NOTIFICATION

No impact

5.2.4.3 PCU COUNTERS

No impact

5.2.4.4 PCUOAM COUNTERS

No impact

5.2.4.5 MISCELLANEOUS

No impact

5.3. TRANSMISSION NETWORK INTERFACES

5.3.1 PCM / OPTICAL INTERFACE

No impact




5.3.2 ETHERNET

No impact

5.3.3 X25

No impact




6. O&M PROCEDURES

6.1. UPGRADES

6.1.1 OMC-R UPGRADE

No Impact

6.1.2 W-NMS UPGRADE

No impact




6.1.3 BSC, PCUSN, TCU & BTS UPGRADE

The robust IN/TCU upgrade protocol is desirable before upgrading to V16 since IN/TCU includes multiple interfaces break. This interface breaks increase the likelihood of long outages due to premature boards reset (see Section 4.7.2)

The IN/TCU load delivery along V16 will provide minimal backward compatibility interface over LAPD and AAL-5 between boards of the critical path. The compatibility is minimal because it applies only to a small number of configuration messages which are mandatory to restore the critical path. The compatibility is backward in the sense it guarantees only that an I/F board (i.e ATM-RM or IEM) in N+1will interoperate with a CEM in N release.

This minimal backward compatibility is expected to be resilient to any node or board failure scenario; that is any failure scenario should never lead to isolate a TCU or IN node from the control node.

The only scenario that is cause for concern is if one CEM has upgraded and after the nodal reset, the CEM with the older load is active and there is a new load running in the passive CEM. If this happens and the older load is V14, it will continue to attempt to recover the passive CEM indefinitely. In this case the upgrade request will be refused (the upgrade relies on passive CEM recovery in V14). This behavior has changed in V15 so that it will ignore the passive CEM and continue to upgrade itself. So if V14 was on the active CEM, you would have to manually reset the passive CEM first and then the active CEM. This would leave the active CEM (after nodal recovery) running the new load. However, here are the conditions that must be satisfied in order to get into this situation (assume CEM 7 active to start):

1. all cards except active CEM 7 have been upgraded to V16 2. active CEM 7 is running V14 3. passive CEM 8 is successfully flashed and has swacted 4. as soon as the swact is done (i.e. before CEM 7 starts to flash), newly active

CEM 8 resets, giving up activity 5. newly active CEM 7 (running V14 with V14 in flash) resets, causing the nodal

reset 6. After the nodal reset mentioned in point 5, the CEM running the v16.0 load will

come up first because of the reduced duration of Critical Path in v16.0, leaving CEM 7 active with V14 in RAM

The window of vulnerability of this failure scenario is expected to be very short since the newly passive CEM update its passive mate flash right after the switch over. Once the CEM flash has been updated even partially a node reset will never elect this CEM as active.

Note that the downgrade from V16.0 does not need any I/F compatibility between board since a downgrade will trigger the robust upgrade protocol running on the CEM.

The minimal backward compatibility must be maintained through post V16 releases until all BSC and TCU on the field have been upgraded to V16.




If the upgrade online is frozen; the recomendations are the following:

• if the freeze probable cause is “permanent freeze state on TMU” (cause number 7) then by triggering resumeUpgrade TGE, the operator has the option of taking Action=fallback OR Action=upgrade. It is recommended to take Action=upgrade in this case.

• For all other probable causes which relates to a hardware faults the operator should replace the faulty hardware that will automatically lead to resume the upgrade

• If the replacement of the faulty hardware is not possible (remote BSC for instance) then the operator has the option of taking action=fallback OR action=upgrade.

o If the failed card has been upgraded (cards typically fail when reset), it is recommended to take Action=Upgrade.

o If the failed card is not upgraded yet, the operator may take Action=fallback which shall unlikely have CP/service impact, or continue the upgrade with the potential for some CP/service impact.

Important Note: Once resumeTGE is used to cancel the freeze state, any future freeze states are ignored - this always has the potential to provide CP/service impact. The operator must assess the progress of the upgrade, and analyze the fault information to make the best decision on what action (upgrade/fallback) to take.

BSC and TCU upgrades procedures and tools are impacted (see § 8.1.5).

6.2. INSTALLATION AND COMMISSIONING

No impact

6.3. MAINTENANCE

No impact

6.4. OPERATION

No impact

6.5. EXTENSION

No impact




6.6. TECHNICAL ASSISTANCE SUPPORT

No impact




7. FIELD INTRODUCTION

7.1. HARDWARE CONSTRAINTS

No impact

7.2. SPARES CONSTRAINTS

No impact

7.3. BSS VERSION INTERWORKING

This feature is only applicable to BSS versions v16.0 and above.

7.4. NSS INTERWORKING

No impact

7.5. SGSN INTERWORKING

No impact

7.6. FUNCTION INTERWORKING

No impact

7.7. ENGINEERING

No impact

7.7.1 METRICS

7.7.2 RULES




8. SUB SYSTEMS IMPACTS

8.1. OAM

8.1.1 OMC-R

No Impacts

8.1.2 W-NMS

No Impact

8.1.3 PCUOAM

No impact

8.1.4 TOOLS

8.1.4.1 CT2000

No impact

8.1.4.2 PERFORMANCE MONITORING TOOL

No impact

8.1.5 NRP/UPGRADES BSS

8.1.5.1 TCU E3 UPGRADE

TCU e3 upgrade NRP tool and procedures should be modified.

If a BSC is V14 or V15, then lock/unlock commands should be used for TCU off-line upgrade and fallback.

If a BSC is V16 or higher, then TCU should not be locked prior to the TCU offline upgrade.

8.1.5.2 BSC E3 UPGRADE

BSS Upgrade Procedure (DS/BSS/APP/0010) should be modified to advertise that




the operator action is needed in case of permanent freeze upgrade state: an operator should choose either to continue upgrade or to fallback by sending “resume upgrade” command through a selected resumeType value. Note that the NRP tool itself will not trigger automatically a resumeUpgrade TGE since it can decide which actions resume or fallback the operator will choose.

8.2. BSC 2G

This feature is not applicable to BSC 2G.

8.3. BSC E3

This feature impacts following applications inside BSCe3:

Control Node software:

o ADM: management of new upgrade status values

o ADM new TCU upgrade backgroundTcuUpgrade and RestoreTcuReleaseConsistency TGE

o ADM: TGE resume upgrade

o BaseOS : management of XX_TM background flashing

o BSP: remove interruptions in flash driver

o UpgradeCN: duplexity check during TMU upgrade online

o UpgradeCN: processing TEA resume upgrade

o UpgradeCN: management of background flashing

o HM: management of background flashing

o UpgradeCN: new values of the upgrade status

o SupCN: new state change due to upgrade status

o SUP_IN: management of new values of the upgrade status state change.

o SUP_TCU: management of new values of the upgrade status state change.

Interface Node Software:

o New values of the upgrade status.

o Reduced critical path latency.

o Robust offline upgrade manager with management of N+1 secondary flash

TCU Node Software:

o New value of the upgrade status state change




o Reduced critical path latency

o Robust offline upgrade manager with management of N+1 secondary flash

8.4. BTS

No impact

8.5. TCU 2G

This feature is not applicable to TCU 2G.

8.6. TCU E3

This feature impacts following applications inside TCUe3:

CEM board: management of 64K switching orders coming from Control Node.

8.7. PCUSN

8.7.1 PCUSN

No impact

8.7.2 PCUSN – PCUSPE2

No impact




8.8. TOOLS

Impact Y/N Comments Maintenance tools N TML platform N (HW, SW, installation procedure…) TML BSC2G application N TML TCU2G application N TIL RTC application N TIL COAM application N TML BSC e3 application N TML TCU e3 application N ROT N RACE N Validation tools N Simulators N SPY N




9. ABBREVIATIONS AND DEFINITIONS

9.1. ABBREVIATIONS

MIB Management Information Base

CN Control Node

IN Interface Node

TCU Transcoder Unit

QUNI Query Useful Node Information

RM Resource Manager

IEM Interface Electronics Module

CEM Common Equipment Module

SRT SubRate Timeswitch

TRM Transcoder Resource Module

OMU Operation & Maintenance Unit

TMU Traffic Management Unit

FT Fault Tolerance

LB Load Balancing

HM Hardware Management

CP Core Process

SupCn Supervision Control Node

UpgradeCn Upgrade Control Node

OMC Operation & Maintenance Center

NRP Network Reconfiguration Procedure

NTPs Nortel Telecom Publications

9.2. DEFINITIONS

Default value:

this value is used:

- either when an OMC-R release upgrade occurs and the value of a new parameter is ["false"] or ["null"] or [x] in order to ensure the same performance before upgrades (for example from V10/V11 to V12).

- or when a customer wants to deactivate a function.




Recommended value:

This value is recommended by Nortel Network for new networks roll out before the optimization. The recommended values take benefit of the performance of Nortel Network BSS feature.

For installed networks, the value to set depends on the chosen options or rules defined by the customer.

END OF DOCUMENT

pebscdd012924e0110

Documents

Transcript of pebscdd012924e0110