August 21, 2006

UMTS Operations

Lucent Worldwide Services

Knowledge Transfer Workshop

Node B Alarm Management

Table of Contents TASK DESCRIPTION ................................................................................................................................... 3 NODE B......................................................................................................................................................... 3

NE Event could not be Processed............................................................................................................. 3 NE Alarm Synchronization in Progress ..................................................................................................... 3 NE Communication Problem ..................................................................................................................... 3 UNMANAGED State Set for NE ................................................................................................................ 3 HeartBeatFailure ....................................................................................................................................... 3

E1T1 PORT................................................................................................................................................... 4 AISDetected .............................................................................................................................................. 4 BitErrorRateGreaterThan1PerMillion ........................................................................................................ 4 SignalLost .................................................................................................................................................. 4 FrameLost.................................................................................................................................................. 4 FrameMissed............................................................................................................................................. 5 IMALinkAlarm ............................................................................................................................................ 5

CTU............................................................................................................................................................... 5 ClockFailure30_72MHz ............................................................................................................................. 5 InitializationFailure..................................................................................................................................... 5 SelfTestFailure .......................................................................................................................................... 5

FRAME.......................................................................................................................................................... 6 CabinetDoorOpen...................................................................................................................................... 6

SHELFFAN.................................................................................................................................................... 6 EquipmentFault ......................................................................................................................................... 6

IUBINTERFACE............................................................................................................................................ 6 ALCAPLinkFailure ..................................................................................................................................... 6 NBAP LINK FAILURE................................................................................................................................ 6

RADIO ........................................................................................................................................................... 7 CommunicationFailure............................................................................................................................... 7 MajorTxSpectrumFailure ........................................................................................................................... 7 MajorGainFailure ....................................................................................................................................... 8 MinorGainFailure ....................................................................................................................................... 8 LossOfPulse .............................................................................................................................................. 8 LossOfForwardBusLink ............................................................................................................................. 8

IMAGROUP................................................................................................................................................... 9 CriticalGroupAlarm .................................................................................................................................... 9 MajorGroupAlarm ...................................................................................................................................... 9 NonCriticalGroupAlarm.............................................................................................................................. 9

FILTER........................................................................................................................................................ 10

Lucent Technologies PROPRIETARY

See notice on cover page

DCConverterFailure................................................................................................................................. 10

Node B Alarm Management August 21, 2006,

RxAmpFailure.......................................................................................................................................... 10 EXTRXAMP................................................................................................................................................. 10

ExtRxAmpFailure..................................................................................................................................... 10 CELL ........................................................................................................................................................... 11

ReceiveAntennaTestThresholdExceeded ............................................................................................... 11 PowerTestThresholdExceeded ............................................................................................................... 11

AMPLIFIER ................................................................................................................................................. 12 OverTemperature .................................................................................................................................... 12 RFOverDrive............................................................................................................................................ 12 CommunicationFailure............................................................................................................................. 12 InitializationFailure................................................................................................................................... 12

RF PATH DIAGRAM ................................................................................................................................... 13 RXAIT EXAMPLE CONFIGURATION ........................................................................................................ 14

Lucent Technologies Proprietary See notice on cover page 2

August 21, 2006, Node B Alarm Management

TASK DESCRIPTION

This section describes Node B alarm management and the meaning of various Node B alarms.

NODE B NE Event could not be Processed

The data sent by the Node B has been corrupted. Fault Clearance:

Verify Node B is Managed Perform a Full Tree Sync NE Alarm Sync

NE Alarm Synchronization in Progress

Typically reported when communication has been restored to the Node B. It is an information alarm and can be ignored. Fault Clearance:

Usually found in the cleared alarm list

NE Communication Problem

Alarm is generated when communication to the Node B over the logical Itf-B link has failed. Fault Clearance:

Verify the Node B is not in the Pre-Provisioned State, indicating that it has not been integrated, however it has been mistakenly managed. If Pre-Provision state exists in the Node B details, Unmanage the Node B to clear the alarm.

Ping the Universal Master Controller (UMC) IP Address. If ping fails troubleshoot connectivity issues.

If ping is successful Unmanage and Manage the Node B. If alarm still exists Perform an NE Alarm Sync.

UNMANAGED State Set for NE

The Node B link state has changed to Unmanaged. Call processing should still be possible; however alarm events reporting trouble will not be visible. Fault Clearance:

If the Node B is in the Pre-Provisioned state ignore the alarm and acknowledge the alarm to let others know the alarm has been verified.

To clear the alarm Manage the previously integrated Node B.

Lucent Technologies Proprietary See notice on cover page 3

HeartBeatFailure The heartbeat synchronization between the Node B and the OMC-UPS has been lost. Fault Clearance:

Verify the Node B is not in Pre-Provision mode. If it is, verify the Node B is Unmanaged and ignore the alarm by acknowledging so that others are aware the alarm has been troubleshot.

Ping the UMC IP address. If ping fails, troubleshoot connectivity. UnManage / Manage the Node B Perform an NE Alarm Sync.

Node B Alarm Management August 21, 2006,

E1T1 PORT AISDetected The Node B has detected an AIS signal (Blue Alarm), which is most likely a failed T1.

Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the Network

Interface Unit (NIU) and Channel Service Unit (CSU). If loops are found and removed verify alarm has cleared and service has been

restored. If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with

the T1 provider to report the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch a tech to resolve.

BitErrorRateGreaterThan1PerMillion

The CSU in the Node B (T1 port) has detected several bit errors, or Cyclic Redundancy Checks (CRCs). The T1 may remain InService; however the link quality will suffer. The bit errors are also called Dirty T1. Fault Clearance:

Request Cingular DACs support to monitor the T1 for at least 15 minutes, or longer if possible. If no trouble is found, request Cingular to break the circuit and stress the circuit (run multiple patterns) to the CSU (but not the NIU).

SignalLost The CSU (Node B T1 port) has detected a loss of signal.

Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the NIU and

CSU. If loops are found and removed verify alarm has cleared and service has been

restored. If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with

the T1 provider reporting the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch a tech to resolve.

FrameLost The Node Bs CSU cannot detect the framing pattern. This is also known as a RED Alarm:

Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the NIU and

CSU. If loops are found and removed verify alarm has cleared and service has been

restored. If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with

the T1 provider reporting the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch a tech to resolve.

Lucent Technologies Proprietary See notice on cover page 4

August 21, 2006, Node B Alarm Management

Continued on next page E1T1 PORT, CONTINUED FrameMissed Node B is detecting a framing error on the T1 line.

Fault Clearance:

Request Cingular DACs support to monitor the T1 for at least 15 minutes, or longer if possible. If no trouble is found, request Cingular to break the circuit and stress the circuit (run multiple patterns) to the CSU (but not the NIU).

IMALinkAlarm A T1 that is a member of the Inverse Multiplexing on ATM (IMA) Group is either OutofService

or is InService but did not join the IMA Group. Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the NIU and

CSU. If loops are found and removed verify alarm has cleared and service has been

restored. If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with

the T1 provider reporting the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch a tech to resolve.

CTU ClockFailure30_72MHz

The Common Timing Unit (CTU) has failed to detect a 30.72MHz clock signal. Call processing is affected because the Node B will not fully initialize. This alarm will typically be found in the cleared alarm list because the CTU and Oscillator Module (OM) will automatically reset to resolve. If the problem that caused the clock failure remains, the CTU will most likely fail Initialization and report this alarm. Fault Clearance:

Verify initial cause for clock failure replace suspect OM. Replace failed CTU.

InitializationFailure

The CTU has failed to initialize properly either during Node B startup or because of an autonomous reset to self-clear an alarm. Fault Clearance:

Check cleared alarms to find out why the CTU re-initialized. Reset the CTU if the reset does not resolve the alarm, replace CTU or OM.

SelfTestFailure The CTU raises the the SelfTestFailure alarm anytime it resets and fails the Board Level Self

Tests (BLST). Fault Clearance:

Reset CTU. If it fails again, replace it. Lucent Technologies Proprietary See notice on cover page 5

Node B Alarm Management August 21, 2006,

FRAME CabinetDoorOpen

The Outdoor Node B (modular/micro) has triggered an alarm indicating the door is open. Fault Clearance:

Dispatch a Tech to the site to verify the door is closed and the alarm contact is connected and unobstructed.

Onsite, verify that the rear heat exchanger hatch is secure and screws are tight.

SHELFFAN EquipmentFault The fan has failed because the power to the fan has been disconnected or the fan is faulty.

Fault Clearance:

Dispatch Tech to site to verify that the fan breaker is on. Replace the faulty FanShelf.

IUBINTERFACE ALCAPLinkFailure

Access Link Control Application Protocol (ALCAP) virtual circuit connection has failed. Fault Clearance:

In the PSAX (Bpsax, and Rpsax), verify the Permanent Virtual Circuits (PVCs) exist and pass 2-way traffic. If the PVCs do not exist, create them. If the PVCs do not pass 2-way traffic, delete and add them back (PSAX Bug).

If the Node B has just rebooted, wait for up to 5 minutes for the alarm to clear. Verify the IuB is not in alarm under the Lrnc.

NBAP LINK FAILURE

Node B Application Part (NBAP) virtual circuit connection has failed. Fault Clearance:

In the PSAX (Bpsax, and Rpsax), verify the PVCs exist and pass 2-way traffic. If the PVCs do not exist create them. If the PVCs do not pass 2-way traffic, delete and add them back (PSAX Bug).

If the Node B has just rebooted, wait for up to 5 minutes for the alarm to clear. Verify the IuB is not in alarm under the Lrnc.

Lucent Technologies Proprietary See notice on cover page 6

August 21, 2006, Node B Alarm Management

RADIO CommunicationFailure

Heartbeat synchronization of the radio has been lost. Call processing is not possible. The following could be reasons that the radio can not communicate:

Faulty Radio Faulty CTU or OM

Fault Clearance:

Reset the radio if the initialization fails, replace the radio and calibrate. If alarms reappear troubleshoot the clock (replace CTU or OM).

MajorTxSpectrumFailure MajorTxSpectrumFailure is a general alarm used for several purposes. A

majorTxSpectrumFailure can be triggered by: Antennas not connected Failure of edpdul Radio hardware failure (known cold soldering issue) Incorrectly connected Rf cables

Fault Clearance:

Execute RxAntennaTest if results are less than 14dB, dispatch an antenna crew (anything close to 0.5db indicates an open or a short in the antenna system).

Telnet to the radio and look for the following messages repeating every couple minutes:

MCRedpdulFsm_generate: port=1 edpdulResult=19 blueResult=0 (Error) MCRedpdulFsm_trainIt: port=1 (Generate error) MCRedpdulFsm_trainIt: port=1 srxCopyCount=3 (Recapturing) MCRedpdulFsm_estimate: port=1 srxCopyCount=3 (Estimating) MCRedpdulFsm_generate: port=1 (Generating) Or

MCRedpdulFsm_attemptToStartTraining: port=1 (Training) {Nothing in NV} MCRedpdulFsm_startTransmission: port=1 session=2128 MCRedpdulFsm_startTransmission: port=1 pwr=31622 (Transmitting, backed off) MCRedpdulFsm_startTransmission: port=1 pwr=39810 (Transmitting, cell power) MCRedpdulFsm_ddcEventHandler: (HOLD complete) MCRedpdulFsm_trainState: port=1 (Unlocked, discarding captures) MCRedpdulFsm_trainState: port=1 clgcGainError=920 MCRedpdulFsm_attemptToStartTraining: port=2 (Not training) {Disabled}

If one of the above messages is repeating, dispatch a Tech to the site to calibrate the radio. If the alarm returns replace the radio.

Verify antenna sharing configuration is correct at the site (div 0 = tx/rx; div 1=rx). Ensure that diversity 1 is not connected to the rx input of a duplexer.

Swap antennas with a known good sector and determine if the problem follows the Node B or the antenna system

Lucent Technologies Proprietary See notice on cover page 7

Node B Alarm Management August 21, 2006,

MajorGainFailure MajorGainFailure is generated when closed loop gain control has failed, and the radio has switched over to OnBoard Gain Control. Fault Clearance:

Execute Power Test if results are greater than 1dB, dispatch a Tech to the site to verify power in closed loop and open loop mode.

Replace radio and calibrate (especially when MajorGainFailure is coupled with a TxSpectrumFailure indication of the known HW cold soldering issue)

MinorGainFailure MinorGainFailure can be caused by the following:

Missing Antenna Path descriptor data Poor previous calibration Node B internal RF cabling Issue

Fault Clearance:

Verify Antenna Path Descriptor data is present in the backplane using remote maintenance terminal (RMT).

Dispatch an Installer to perform power calibration. LossOfPulse LossOfPulse can occur under the following conditions:

High temperature Rx Signal Too strong (numerous Rx Blocks) Faulty CTU Faulty OM

Fault Clearance:

Verify the alarm is present on all 3 radios if yes replace OM or CTU Telnet to the Multi-Carrier Radio (MCR) and type: tempSenseStats. If the radio

reports anything higher than 55 degrees C, report over-temperature condition to Cingular Operations Team.

Dispatch Tech to the site to swap radios. If the problem followed the radio replace the radio; else return the radio and troubleshoot antenna path and internal Node b cables.

If the alarm appears with the antennas connected, and does not appear when antennas are disconnected, dispatch antenna crew; else verify the internal Node B cabling (recommend power calibration or swapping filters).

LossOfForwardBusLink

One of the Field Programmable Gate Arrays has failed on the radio. The radio will autonomously reset and perform BLST. If BLST passes, the radio will return to service; else an alarm will be raised indicating Initialization Failure and will require the radio to be replaced. FaultClearance:

This alarm is typically found in the alarm history and cleared alarm list. Verify the radio has recovered; if not, replace radio and calibrate.

Lucent Technologies Proprietary See notice on cover page 8

August 21, 2006, Node B Alarm Management

IMAGROUP CriticalGroupAlarm

The IMA Group is no longer Operational. Most likely, all T1(s) have failed. Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the NIU and

CSU. If loops are found and removed verify alarm has cleared and service has been

restored. If the Network Interface Unit (NIU) (Demarc / smartjack) is not able to loop up,

Cingular will open a ticket with the T1 provider to report the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch a Tech to resolve

MajorGroupAlarm

The IMA Group remains operational; however it has lost one or more of the T1s associated with the IMA Group.` Fault Clearance:

Verify the alarm in the PSAX. Request Cingular DACs support to check for any loops, and to loop the NIU and

CSU. If loops are found and removed verify alarm has cleared and service has been

restored. If the NIU (Demarc / smartjack) is not able to loop up, Cingular will open a ticket with

the T1 provider to report the outage. If the NIU loops but the CSU does not, there is problem with the cabling between the

NIU and CSU (Node B) dispatch tech to resolve

NonCriticalGroupAlarm

A NonCritical Configuration problem could indicate the following failure(s): T1 outage IMA GroupID does not match what is provisioned in the NDP on the OMC vs the

PSAX PVCs blocked

Fault Clearance:

Verify the T1/IMA status and IMA configuration in the PSAX and compare that to what is configured in the OMC.

Lucent Technologies Proprietary See notice on cover page 9

Node B Alarm Management August 21, 2006,

FILTER DCConverterFailure

The DC converter inside the filter has detected a failure. Typically this alarm is caused by someone onsite who has replaced a failed filter and has turned off or disconnected power to the filter. Fault Clearance:

Verify filter shelf breakers. Verify DC cable is secure.

RxAmpFailure RxAmp Failure can be caused by the following:

Lightning Faulty filter DC power/Comm cable issue

FaultClearance:

Dispatch a tech to the Site to reset the filter shelf. If filter remains in alarm replace filter and calibrate sector.

EXTRXAMP ExtRxAmpFailure When the Node B provides the 12VDC power to the Tower Top Low Noise Amplifier (TTLNA)

(extRxAmp), the extRxAmp managed object should appear in the OMC GUI and should report alarms based on the TTLNA Descriptor data for the Node B input during integration. If the Filter Panel exceeds the limits specified, the DC bias powering the TTLNA will be turned off, and the RF coverage area will shrink. Fault Clearance:

Verify the TTLNA descriptor data is populated in the Filter Panel (note that any time a filter is replaced, the TTLNA data must be re-entered and Node B reset in order for changes to take effect).

Using the RMT, sample the current reading of the TTLNA. If it is out of the alarm range, dispatch a Tech to troubleshoot further onsite.

Onsite request a Tech to swap antenna 0,1. If the problem followed the antenna dispatch the antenna crew to resolve. If the problem followed the filter- replace the filter and calibrate the associated sector.

Lucent Technologies Proprietary See notice on cover page 10

August 21, 2006, Node B Alarm Management

CELL ReceiveAntennaTestThresholdExceeded

RxAntennaTest is a useful tool to measure the Return Loss of the antenna system. However, it has some limitations to be aware of:

1) Antenna test cannot sweep past TTLNA if the filter is not powering the TTLNA. The path from the Node B External Antenna Connection (EAC) to the TTLNA Input will be tested, but not beyond.

2) RxAntenna test cannot sweep past an RxAIT antenna-sharing configuration. RxAIT splits the diversity into separate tx/rx paths. Anything beyond the RxAIT equipment will be untested.

FaultClearance:

RxAntennaTest alarms indicate that the antenna test has failed and an antenna crew should be dispatched to resolve issue.

Note: The RxAntennaTest could pass and indicate Test Unit Not Available. If this message occurs or the RxAntennaTest Times Out Unmanage/Manage the Node B and try again.

PowerTestThresholdExceeded

Power Test is a useful tool to measure instantaneous Peak RF Ouput power at the Node B Equipment Antenna Connector (EAC) without having to send someone to the site to verify. Typically for a 40W configured system, the power output range should be between 36 and 38 dBm. The power test will fail if the expected power and the detected power differ by more than 1dB (aka Err DET or Results). Fault Clearance:

Execute 3 or more consecutive power tests. If all the results are consistently within the 36-38dBm range, but the results are above the 1dB threshold, dispatch someone to the site to perform calibration on the associated sector.

If the test results pass/fail with peak measurements less than 15dB, verify the logical cells are enabled, and that Common Pilot Channel (CPICH) and Common Channel Configuration is configured on the Node B.

If the test results pass/fail with results greater than 40dBm, dispatch a Tech to the site to verify Power in Open Loop and Closed Loop mode to determine if the amplifier and radio are operating properly (typically the failed radios is outputting messages, indicating that edpdul is training every couple of minutes. Also, a MajorTxSpectrum alarm may be present. Check / troubleshoot radio alarms).

Lucent Technologies Proprietary See notice on cover page 11

Node B Alarm Management August 21, 2006,

AMPLIFIER OverTemperature The amplifiers temperature sensor has detected a temperature above the specified limit.

Fault Clearance:

If the alarm exists on all amplifiers dispatch a Tech to the site to verify the fans are operating and the ambient temperature is within normal operating ranges.

Reset the amplifier if the alarm continuously reappears/clears replace amplifier.

RFOverDrive The amplifier has detected an overdrive condition that could be caused by one of the

following: Excessive Transmitter output Faulty calibration Failed Radio Failed Amplifier

Fault Clearance:

Reset the amplifier if the alarm re-appears reset the radio. Dispatch a Tech to the site to calibrate this sector. Suspect Radio replace radio and calibrate. Suspect Amplifier replace amplifier and verify RF power.

CommunicationFailure

Communication failure on the amplifier can be caused by the following: Faulty I2C cables Faulty PDPII No Power to amplifier Faulty amplifier

Fault Clearance:

Inspect all circuit breakers, cables. Power cycle the amplifier circuit breaker. Verify RF Power Closed Loop. Replace amplifier (ensure power is off when RJ45 cable is plugged into amplifier

/Power Distribution Panel (PDP), or else the amp could go into a bench test mode power cycle will resolve).

Perform a continuity check on the cables to ensure they are correctly wired and completed terminated.

Replace PDP.

InitializationFailure

Amplifier has established communication to the Controller. However, the amplifier has failed to initialize completely. Fault Clearance:

Reset Amplifier. If no change replace amplifier and perform closed loop power verification.

Lucent Technologies Proprietary See notice on cover page 12

August 21, 2006, Node B Alarm Management

RF PATH DIAGRAM

STAT

Tx1-ON

TX 2-On

RC 1

Rx 1

Rx 2

RC 2

RF OUT

ALM

ACT

RF IN

Lucent

J8 J28

Luce

nt

J11

J31

J6J5

J26

J25

J7

Dx0 J

4

J24

J2-C

J22-

CJ4

1

Dx1

RF-

ON

-0LN

A-0 LN

A-1

TTLN

A-0

RF-

ON

-1TT

LNA

-1

B2

(RX

1)

B2

(RX

2)

MCR

AMP

Filter

J4

RX1

RC 1

J 24

RX2

Ant

enna

Tes

t

J 41

Tone

B3

Tx1

J8

B1

J 11

Tx On

Tx Off

Tx1

Tx2

Tone

MCR

Lucent

Lucent Technologies Proprietary See notice on cover page 13

Node B Alarm Management August 21, 2006,

RXAIT EXAMPLE CONFIGURATION

Lucent Technologies Proprietary See notice on cover page 14

Node B Alarm ManagementTASK DESCRIPTIONNODE BNE Event could not be ProcessedNE Alarm Synchronization in ProgressNE Communication ProblemUNMANAGED State Set for NEHeartBeatFailure

E1T1 PORTAISDetectedBitErrorRateGreaterThan1PerMillionSignalLostFrameLostFrameMissedIMALinkAlarm

CTUClockFailure30_72MHzInitializationFailureSelfTestFailure

FRAMECabinetDoorOpen

SHELFFANEquipmentFault

IUBINTERFACEALCAPLinkFailureNBAP LINK FAILURE

RADIOCommunicationFailureMajorTxSpectrumFailureMajorGainFailureMinorGainFailureLossOfPulseLossOfForwardBusLink

IMAGROUPCriticalGroupAlarmMajorGroupAlarmNonCriticalGroupAlarm

FILTERDCConverterFailureRxAmpFailure

EXTRXAMPExtRxAmpFailure

CELLReceiveAntennaTestThresholdExceededPowerTestThresholdExceeded

AMPLIFIEROverTemperatureRFOverDriveCommunicationFailureInitializationFailure

RF PATH DIAGRAMRXAIT EXAMPLE CONFIGURATION