AR5

New AR: View AR | AR without Proprietary

History | States | Assignment | Timetracking

Add Attachment

AR Text Search | More | CARES HomeAssistance Request 1-5877056

Customer Ticket N/A

Short Loss of supervision on RNC235 and NodeBs under it

Description

Current Aug 24 : Analysis provided from the available logs, Issue was

Summary caused due to CP card crash from slot-1, recommended to replace

the board in maintenance window. Closure confirmation received

to close this AR.

State Closed : Solution/Service Provided

Outage No

Severity 1

Priority 1

Service Request Remote Technical Support

Request Type Support

Sub-Type Diagnose

Category Other

Sub-Category Customer Dedicated Support

Internal Yes

Assignment Events

Ask Alcatel-Lucent

Product

Product 9370 RNC

Version LR13.3.W

Patch or SU LR13.3.1

No Change Request is linked to this AR

Product Location

Site ETISALAT ABU DHABI

Instance 9370RNC-Etisalat : In Service

Site Company Emirates Telecommunications Corporation

City Abu Dhabi

Country United Arab Emirates

Contact

Name Karim IBRAHIM ABD EL NABY

Company Alcatel-Lucent : Open on behalf of Customer

Phone 971 56 354 4568 Email [email protected]

Request Method Email-Eng

Dates

Occurred 15-Aug-2015 09:58 Time now 29-Aug-2015 09:21 (GMT+4)

Reported 15-Aug-2015 09:58 AR Created 15-Aug-2015 10:01

Service Start 15-Aug-2015 09:58

Next Customer Contact 27-Aug-2015 14:30

Responded 15-Aug-2015 10:09 Respond Target 15-Aug-2015 10:28 SA - Calculated

Restored 15-Aug-2015 12:10 Restore Target 16-Aug-2015 13:36 SA - Calculated

Resolved 19-Aug-2015 08:10 Resolve Target 23-Sep-2015 04:27 SA - Calculated

Targets for Restore & Resolve were extended by 8 days 18 hours

to account for Pending time

Closed 24-Aug-2015 14:09

Last Modified 24-Aug-2015 14:09 Modified By pthota

Entitlement

Agreement 242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script This request is entitled to service.

People

Owner TSCd-WLS-WCDMA-IN : pthota

Assignee CloQ-WLS-WCDMA-vGlobal : pthota

Referred 1 NorP-WLS-WCDMA-vGlobal : pthota

Resolve Group NorP-WLS-WCDMA-vGlobal

Submitter igawrych

Description

Attachments

Loss of supervision on RNC235 and NodeBs under it

Showing Investigation and Proprietary logs together in time order Show separately

Investigation Log and Proprietary Log

1. 15-Aug-2015 12:30 pthota

Update to Current Summary: Aug 15 : Under Investigation

2. 15-Aug-2015 12:31 pthota

Update to Current Summary: Aug 15 : Awaiting Receive Log Files / Technical Info

3. 15-Aug-2015 12:34 ALCATEL-LUCENT PROPRIETARY pthota

On 15.08.15 07:57, Karim (Karim)** CTR ** IBRAHIM ABD EL NABY wrote:

Hello

PROBLEM DESCRIPTION: Loss of supervision on RNC235 and NodeBs under it

PROBLEM IMPACT:

Loss of supervision on RNC235 and it's NodeB, the history as below

1- While doing unlock for Fddcell under RNC235 it take a very long time about

2 hours and not locked

2- Stop/Start for the pmgt_cp_utran WMS process done, and all process come up,

3- After that we observe loss of supervision on both RNC235 and it's NodeBs,

EXPECTED RESULTS:

RCA and Soln for this issue as the customer is escalating us due to this issue,

Contact Surname : Karim

Contact Given Name : Ibrahim

Contact Phone :+971 56 354 4568

Contact Company : Alcatel-Lucent

Company : Emirates Telecommunications Corporation

Contract Number : 242895

Region : EMEA

Country : United Arab Emirates

City : Abu Dhabi

Site : ETISALAT ABU DHABI

Product Instance : 9370 RNC

Severity : 1

Priority : 1

Product Version : LR13.3.1

Outage : No

Internal/External: :Internal

Thanks,

BR,

Karim Ibrahim

LTE/3G Integration Professional

Alcatel-Lucent, Dubai, Etisalat Program

M: +971(0)563544568| ONNET: 22341798

Every Success has its network

From: [email protected] [mailto:[email protected]]

Sent: Saturday, August 15, 2015 10:24 AM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Cc: WAGDY MANSOUR, AYMAN (AYMAN)** CTR **; KHEDR, MAHMOUD (MAHMOUD); ALI, YASSER

(YASSER)** CTR **; ABDEL-HALIM, SAYED (SAYED); MERAD, ABDESSAMAD (ABDESSAMAD);

EL-MIDANY, AHMED (AHMED)

Subject: Re: Etisalat UAE//Loss of supervision on RNC235 and NodeBs under it /

1-5877056

Dear Customer,

Thank you for contacting the Alcatel-Lucent Welcome Center.

Your request has been processed and the ticket number for your reference is AR

1-5877056

Please reference that number when contacting Alcatel-Lucent for follow-up.

An Alcatel-Lucent representative will contact you regarding your request.

Kind Regards,

Izabela Gawrychowska

ALCATEL-LUCENT

GLOBAL WELCOME CENTER

Visit OnLine Customer Support for Contact details

From: [email protected]

[mailto:[email protected]] On Behalf Of IBRAHIM

ABD EL NABY, Karim (Karim)** CTR **

Sent: Saturday, August 15, 2015 12:39 PM

To: INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **



EL-MIDANY, AHMED (AHMED)

Subject: RE: Etisalat UAE//Loss of supervision on RNC235 and NodeBs under it /

1-5877056

Hi

Please update who is assigned to this AR.

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798



[mailto:[email protected]] On Behalf Of THOTA,

Praveen-K (Praveen-K)** CTR **





EL-MIDANY, AHMED (AHMED); INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **

Subject: AR 1-5877056//Critical//9370 RNC//LR13.3.1//Etisalat UAE//Loss of

supervision on RNC235 and NodeBs under it

Hi Karim,

I am the assignee of this AR.

As discussed issue resolved now with CP switchover action performed from the RNC.

Please confirm.

Please provide below details and logs for further investigation of this issue.

CONFIGURATION DETAILS :

1. Hardware configuration:

2. Software configuration:

>OAM:

> RNC: LR 13.3.1

> NodeB:

HISTORY:

1. Which & how many NEs impacted ? Only One RNC 235 and its NodeBs...Please

confirm.

2. When the issue has started?

3. What actions were performed before or in time when the issue begun (feature

activation, upgrade, 3rd party actions...)

4. What recovery actions have been done & with what results: CP switchover

done and issue resolved --- Please confirm.

PROBLEM IMPACT:

1. Frequency of the problem (once, x times, sporadic, systematic):

2. Impact (drop call, board reset, KPI degradation, redundancy, None):

LOGS REQUIREMENT:

1) HFB of the impacted RNC before 5 days from issue to till time.

2) RNC Log collection full for the impacted RNC

3) RNC Health check of the impacted RNC.

4) Please provide below commands output from the RNC.

os pcsShowCards

os excDumpCheck

d fs

d fs part/*

d sh ca/* spserv

d sh ca/* carduptime

Thanks and Regards,

Praveen Thota

+91 96899 18050

4. 15-Aug-2015 12:58 ALCATEL-LUCENT PROPRIETARY nrehan

Incoming E-mail from Praveen-K (Praveen-K)** CTR ** THOTA

Workflow Status: Handled by an agent


1-5877056

Contact: Praveen-K (Praveen-K)** CTR ** THOTA

From: "THOTA, Praveen-K (Praveen-K)** CTR **"

<[email protected]>

To: "GAWRYCHOWSKA, Izabela (Izabela)** CTR **"

<[email protected]>, ALU Support

<[email protected]>

Cc: INDIA-WCDMA-UTRAN-BE <[email protected]>, "AGRAWAL,

Ritesh (Ritesh)** CTR **" <[email protected]>, "ARORA,

Gaurav (Gaurav)** CTR **" <[email protected]>

Sent on: August 15, 2015 2:09:42 PM IST

Hi Welcome Center/Izabela,

Could you please let me know the reason for not calling me for this AR assignment?

I have not received any call for this critical AR.

I was just going through my mails and found that one Critical AR was assigned to

me without any information to me.

Thanks and Regards,

Praveen Thota

+91 96899 18050

5. 15-Aug-2015 13:01 ALCATEL-LUCENT PROPRIETARY sunitap

Incoming E-mail from Izabela (Izabela)** CTR ** GAWRYCHOWSKA

Workflow Status: Handled by an agent


1-5877056

Contact: Izabela (Izabela)** CTR ** GAWRYCHOWSKA

From: "GAWRYCHOWSKA, Izabela (Izabela)** CTR **"

<[email protected]>

To: "THOTA, Praveen-K (Praveen-K)** CTR **"

<[email protected]>, ALU Support

<[email protected]>

Cc: INDIA-WCDMA-UTRAN-BE <[email protected]>, "AGRAWAL,

Ritesh (Ritesh)** CTR **" <[email protected]>, "ARORA,

Gaurav (Gaurav)** CTR **" <[email protected]>

Sent on: August 15, 2015 2:22:46 PM GMT+05:30

Dear Sir,

I called you 3 times on the number which was provided both as your office and

mobile number +91 9689918050.

I have followed all instructions for your workgroup, finally I reached Mr. Babu

Sudhakar, the Workgroup Leader, who told me to assign this request to primary from

schedule (you) and give you a call. Unfortunately I could not reach you, so I

assigned it as for Mr. Babu approval.

If you need further investigation, please let me know so I can involve my Team

Leader.

Izabela Gawrychowska

Global Welcome Center

COO – Service Assurance Operations

Alcatel-Lucent

Visit OnLine Customer Support for contact details.


From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **


To: THOTA, Praveen-K (Praveen-K)** CTR **




Subject: RE: AR 1-5877056//Critical//9370 RNC//LR13.3.1//Etisalat UAE//Loss of


Hello Praveen

Please note the issue is recovered by doing lp/o switchover for the RNC , but now

we need to check and find the RC urgently as the customer is pushing ,

The issue started as below

1- Doing unlock for one Fddcell on the RNC235 take about 2 hours without success

2- Stop/start for pm_CP_UTRAN process done on the WMS, and finish successfully

3- After that loss of supervision for RNC235 and all NodeBs under it

4- Lock/Unlock for the OAM link not help to recover the issue

5- Then we go for lp/0 switchover on the RNC

Logs @/home/wcdma/Etisalat_UAE/1-5877056

My comments below and waiting your feedback,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798


From: THOTA, Praveen-K (Praveen-K)** CTR **






Subject: AR 1-5877056//Critical//9370 RNC//LR13.3.1//Etisalat UAE//Loss of


Hi Karim,

I am the assignee of this AR.

As discussed issue resolved now with CP switchover action performed from the RNC.

Please confirm.

Please provide below details and logs for further investigation of this issue.

CONFIGURATION DETAILS :

1. Hardware configuration:

2. Software configuration:

>OAM:LR14.2.1

> RNC: LR 13.3.1

HISTORY:

1. Which & how many NEs impacted ? Only One RNC 235 and its NodeBs...Please

confirm. Yes only RNC235 and it's NodeBs

2. When the issue has started? 6am this morning

3. What actions were performed before or in time when the issue begun (feature

activation, upgrade, 3rd party actions...) mentioned above,

4. What recovery actions have been done & with what results: CP switchover

done and issue resolved --- Please confirm. Mentioned above

PROBLEM IMPACT:

1. Frequency of the problem (once, x times, sporadic, systematic): once,

2. Impact (drop call, board reset, KPI degradation, redundancy, None): loss

of Supervision, during the lp/0 switchover we have about 5 min outage reported

from core team,

LOGS REQUIREMENT:


2) RNC Log collection full for the impacted RNC

3) RNC Health check of the impacted RNC.

4) Please provide below commands output from the RNC.

os pcsShowCards

os excDumpCheck

d fs

d fs part/*

d sh ca/* spserv


Thanks and Regards,

Praveen Thota

+91 96899 18050

7. 15-Aug-2015 16:11 pthota

Update to Current Summary: Aug 15 : Logs download in progress from the server.

8. 17-Aug-2015 09:11 pthota

Update to Current Summary: Aug 17 : Logs received from server, under investigation.










Hello Praveen

Below my comments, and attached the output of the commands, I think something NOK

for all card!,

Wating your feedback,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798










Hi Karim,

I have started downloading the logs, Please note that logs download is taking time

due to the slow connection of data card. I will let you know once RNC log

collection download completed.

[ KARIM] No issue take your time

From RNC health check we could see that --> 1140 fddCells Locked, Could you please

let us know the reason for keeping these many FDDCells in locked state?

[ KARIM] this created dummy sites for prepartions,

Is there any planned event or any other actions done before Fddcell unlock action?

What is the reason for doing fddcell unlock action?

[ KARIM] no any event, and it's normal action,

Did you also perform unlock on any other FDDCell from any other RNCs at that time?

If performed what were the results?

[ KARIM] yes and It was working fine

Could you please let us know the exact time of CP switchover to understand the KPI

degradation from KPI reports, in normal condition CP switchover should not cause

any outage to call processing?

[ KARIM] about 10am, no KPIs degradation happened as it's 15min based from core

see attached they observed 5 min outage at the same time after the lp/0

switchover,

Also if you can please share the communication from core for the 5 minutes outage

to understand the impacted KPI related to this action or not?

[ KARIM] Attached,

Did you try any other actions from WMS point of view before performing CP

Switchover on RNC other than one pm_CP_UTRAN process stop/start? Like all

processes stop/start on WMS?

[ KARIM] Yes we try also stop/start fro all,

Is there any recommendation from support to stop/start the WMS process pm_CP_UTRAN

for recovering the Fddcell unlock command? We want to know the reason for doing

this action since it caused the RNC and it's nodeBs to Loss of Supervision.

[ KARIM] No recommendation, we take the action to try recover the issue and we

already faced on other NWs,

I just gone through the provided HFB, I think you have provided the entire network

HFB, please note it will be very difficult to filter for this RNC in this format.

Request you to extract the HFB for only this RNC from WMS HFB browser window as

per below requirement and provide the same. Thanks for your help.

[ KARIM] I cannot provide per RNC due to the limitation of the number of alarms,

so please use the file I send as it contains all types of alarms,


4) Please provide below commands output from the RNC.[ KARIM] attached,

os excDumpCheck


Thanks and Regards,

Praveen Thota

+91 96899 18050


Sent: Sunday, August 16, 2015 1:10 PM







Hello Praveen

Please update the progress here,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798





Sent: Monday, August 17, 2015 10:32 AM







Hi Karim,

I am going through the logs and will let you know the findings very soon. Thanks.

Thanks and Regards,

Praveen Thota

+91 96899 18050

10. 17-Aug-2015 15:12 pthota

Update to Current Summary: Aug 17 : Analysis provided from the available logs,

requested few more logs and details. waiting for those details.





Sent: Monday, August 17, 2015 3:25 PM




EL-MIDANY, AHMED (AHMED); INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **;

INDIA-WCDMA-WMS-NPO-BE



++

Hi Karim,

Please find below investigation results from the provided logs,

From the HFB I could see below LOS alarms for 14th and 15th 2015 ...

From the above it is clear that issue is not only with the RNC 235, the LOS was

observed on 239,232 and 231 and also WCE RNCs 241 and 242 as well. I could see

that LOS happened on almost all RNCs for two times.

From the above it is suspected that there is some communication issue for the four

RNCs (235,231,232 and 239). which was observed on 14-08-2014 at 17:06 PM and

15-08-2015 at 09:35 AM.

The alarms observed on 15-08-2015 at 09:08 AM for all RNCs including WCE RNCs may

be due to the processes stop and start in WMS -' Could you please confirm at what

the WMS processes stop/start done?

LOS observed on RNC 235 on 15-08-2015 at 04:55 AM was due to TRAP with Disk error

issue on shelf Card/1 as mentioned below.

LOS observed on RNC 235 on 15-08-2015 at 09:40 AM, may be due to manual CP

switchover done at site which need to confirmed from logs.

From the provided RNC 235 RNC Log collection, which was collected at 08:15 AM of

August 15th, I could see the alarms till 2015-08-15 06:13:09.61 only.

For the LOS happened between 09:00 AM 10:00 AM, there is no logs for the

investigation. So for further investigation I request you to provide the below

logs. Kindly upload them to the server with today's date folder. I will let you

know if any further logs required.

1) RNC Log collection full for this RNC ' Please collect it again and upload it

to the server.

2) Also request you to provide HFB for this RNC for 15th August 2015. (if not

please upload the HFB file for August 15th date to the server).

3) /opt/nortel/data/cmXML/CMSessions/YYYYMMDD/Online+Commands/*

4) /opt/nortel/logs/wtk/JobHistory-00114*

5) Screenshot of the unlock command struck in WMS, if not available you can take

it from WMS commands history.

Also as issue happened due to below actions and it is observed on almost 4 RNCs,

we may need to investigate it from WMS point of view as well. Adding WMS support

colleagues to this mail chain for their understanding and support.

1- Doing unlock for one Fddcell on the RNC235 take about 2 hours without success

2- Stop/start for pm_CP_UTRAN process done on the WMS, and finish successfully

3- After that loss of supervision for RNC235 and all NodeBs under it

From below observations and Crash of the Card/1, we recommend you to replace the

Shelf Card/1 with new card in maintenance window and observe the card for next few

days.

I have not observed any KPI degradation from the provided KPI reports of the RNC,

So please provide the specific KPI reports in which degradation is observed.

From the PPC Alarms below are the observations...

As per the alarms I could see that CP Outage happened at 2015-08-15 04:54:52 (CP

outage) due to which the RNC went to Loss of supervision and recovered at

2015-08-15 04:56:11.65

Please find below explanation with alarms...

CP outage happened at below time....

Lp/0 ; 2015-08-15 04:55:07.43

MSG warning equipment 70120203

ADMIN: unlocked OPER: enabled USAGE: active

AVAIL: PROC: CNTRL:

ALARM: STBY: notSet UNKNW: false

Id: 15

Com: Active CP last time in service: 2015-08-15 04:54:52 (CP outage)

Int: 0/1/2/24554; pcsCpElector.cc; 2749; RID3730

After that shelf card/0 is trying to become active...as below..

Shelf Card/0 ; 2015-08-15 04:55:03.35

MSG warning qualityOfService 70120102


AVAIL: PROC: CNTRL:


Id: 2

Com: Active CP starting up on card 0

Int: 0/1/2/24556; pcsCpStarter.cc; 1141; RID3730

OMU-AP reset happened due to CP-swact detected....

Lp/5 Ap/5 ; 2015-08-15 04:55:04.37

SET major equipment 70701005

ADMIN: unlocked OPER: disabled USAGE: idle

AVAIL: PROC: reporting CNTRL:

ALARM: major STBY: notSet UNKNW: false

Id: 5000008

Com: OMU -- AP will be reset; CP swact detected. Resetting Active or Uninitialized

OMU.

Int: 0/0/0/114; apcsApcMgr.cc; 2899; RID37300E1953

After this CP-0 has become active as below from RNC Health check...

###################################################################

### RNC235 STATE CHECKER --- 15/08/2015 at 07:46:07

### -------------------------------------------------

--- RNC Cards list ---

--> Passport's card list : | CP : 2 | PSFP: 12 | OC3: 0 | 4pGe: 2 |

--> RNC overload status is ok

CARD TYPE FSM STATE VISIBILITY LP INSTANCE CPU Util %

---- ---------- ----------------- ---------- -- -------- ----------

0 CPeE RunningAsActive OnLine 0 Active 1

1 CPeE RunningAsStandby OnLine 0 Standby 3

There is a TRAP with Disk Error observed on CP-1 at the same time, due to which

this CP Outage has happened.

Shelf Card/1 ; 2015-08-15 04:56:10.59

MSG indeterminate equipment 70120101


AVAIL: PROC: CNTRL:


Id: 1000001

Com: TRAP DATA INFORMATION:

Unrecoverable error at line 1715 in file sfsFileSys.cc during task

"taccTestCronTask" (0x1b7f9ce0)

--- Reason ---

Disk error, switch over to standby

[sfs:disk test]

0�

Error Info: (0.18)

*** An exception has occurred at task level ***

Offending task "taccTestCronTask" (0x1b7f9ce0) was at address 0x019cf508

(errno = 0x029a000a)

0�

Description Info: (0.17)

Problem is diagnosed as a(n) System Call exception

No handler for exception specific data installed for vector 0xc00

0�

CPU Utilization Info: (0.18)

3 percent busy. CPU: PowerPC750GX D1.2 @ 990MHz (165Mhz 60x bus clk)

0�

Register Info: (0.74)

r00: 019cf504 r01: 1b7f9810 r02: 00000000 r03: 00000032

Int: 1/0/2/13293; pcsCardAgt.cc; 3653; RID3730

Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000002


r04: 1b7f983b r05: 0000000f r06: 00000000 r07: 0000005b

r08: 00000001 r09: 1b7f984a r10: 00000001 r11: 00000003

r12: 40000048 r13: 00000000 r14: 00000000 r15: 00000000

r16: 00000000 r17: 00000000 r18: 00000000 r19: 00000000

r20: 00000000 r21: 1e7f0000 r22: 019f0000 r23: 019f0000

r24: 1e7f0000 r25: 01bb0000 r26: 01bb0000 r27: 00007530

r28: 00000000 r29: 01ba0000 r30: 01ba4d44 r31: 1e7e9bbc

msr: 0000b131 cr: 40000048 ctr: 00000000 pc: 019cf508

xer: 00000000 dar: 00000000 dsisr:00000000 fpcsr:00004000

Interrupt Enable Reg: 000000a0

Link Register: reportDiskError__13SfsFileSystem10DiskIdTypePc+f0 (0x019cf504)


Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000003


MSR[EE]=1 interrupts unlocked

0�

Code Disassembly Info: (67.87)

0x19cf4ec 38a54d6c addi r5,r5, 0x4d6c (0x01ba4d6c)

0x19cf4f0 7fc6f378 or r6,r30,r30

0x19cf4f4 38610008 addi r3,r1, 0x8

0x19cf4f8 38800200 li r4, 0x200

0x19cf4fc 4cc63182 crxor crb6,crb6,crb6

0x19cf500 480a010b bla snprintf

0x19cf504 44000002 sc

-> 0x19cf508 389d4ce8 addi r4,r29, 0x4ce8

0x19cf50c 386006b3 li r3, 0x6b3

0x19cf510 38a10008 addi r5,r1, 0x8

0x19cf514 4cc63182 crxor crb6,crb6,crb6

0x19cf518 4809487b bla Sys_unrecoverableErrorPpc_v


Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000004


0x19cf51c 8001021c lwz r0, 0x21c(r1)

0x19cf520 83a1020c lwz r29, 0x20c(r1)

0x19cf524 7c0803a6 mtspr LR,r0

0�

Stack Trace Info: (3.82)

019cf508 reportDiskError__13SfsFileSystem10DiskIdTypePc+f4 : 019cf508

1b7f9810: 1b7f9a28 019cf51c 6b736944 72726520 *(.......Disk err*

1b7f9820: 202c726f 74697773 6f206863 20726576 *or, switch over *

1b7f9830: 73206f74 646e6174 5b0a7962 3a736673 *to standby.[sfs:*

1b7f9840: 6b736964 73657420 30005d74 63726420 *disk test].0 drc*

1b7f9850: 6420323d 3d636672 000a2031 00000000 *=2 drfc=1 ......*

1b7f9860: 00000000 00000000 1b7f9a50 1b7f9a58 *........P...X...*


Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000005


1b7f9870: 1b7f9a80 1b7f9b38 1b7f9b58 1b7f9b88 *....8...X.......*

1b7f9880: 1b7f9bb8 1b7f9be8 1b7f9c10 1b7f9c18 *................*

019d1950 sfsHandleLocalDiskAccessError+3c0: reportDisk

1b7f9a20: 1b7f9c40 019d1950 * @...P...*

1b7f9a30: 00154e2a 001f0000 00000000 00000000 **N..............*

1b7f9a40: 001e0000 0001b131 1b7f9a80 8538c299 *....1.........8.*

1b7f9a50: 00000005 002853c4 00154e2a 001f0000 *.....S(.*N......*

1b7f9a60: 00000000 1947a720 00000000 1947a720 *.... .G..... .G.*

1b7f9a70: 00000080 0015452b 001ed4d8 00000000 *....+E..........*

1b7f9a80: 1b7f9b38 00095620 00000001 0015452b *8... V......+E..*


Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000006


1b7f9a90: 1b7f9ab8 00000000 1b7f9ac0 00001000 *................*

1b7f9aa0: 1b7f9ac0 00001000 *........ *

019cf25c localDiskAccessTest__13SfsFileSystem+54 : sfsHandleL

1b7f9c40: 1b7f9c50 019cf25c 1e7f0000 00007530 *P...\.......0u..*

019e9f18 sfsAccTestCronTask__Fv+cc : localDiskA

1b7f9c50: 1b7f9c88 019e9f18 eeeeeeee 00000000 *................*

1b7f9c60: 00000000 00000000 00000000 00000000 *................*

1b7f9c70: 00000000 00000000 00000000 00000000 *................*

1b7f9c80: 00000000 019e9e4c *....L... *

000d6768 vxTaskEntry +60 : sfsAccTest

1b7f9c80: 1b7f9ca0 000d6768 * ....hg..*


Shelf Card/1 ; 2015-08-15 04:56:10.59



AVAIL: PROC: CNTRL:


Id: 1000007


1b7f9c90: 016cda40 1ea


Below are the observations from crash report of card CP-1...



--- Reason ---


Error Info: (0.18)




Reset Info: (0.17)

No significant reboot reason available

Last boot type: (warm autoboot)

Disk Error Counters:

ERR-bit1:1 ERR-bit2:1 ERR-bit6:1

B_COR:1

semTake_timeouts:2

srrc=10 drc=2 drfc=1

Serial ID: 13290946F612

Product: Micron_M500_MTFDDAK120MAV

2 disk error record(s) found in Flight Recorder and latest ones shown below

DISK ERROR32 15/08/15 00:54:25, SN:13290946F612, PI:Micron_M500_MTFDDAK120MAV ;

OP:e7 ALT:d0 ERR:0 LBA:a0000200 DAT:0 SC:0 IS:0; stTo:1;

DISK ERROR5 15/08/15 00:54:51, OP:c8 ALT:46 ERR:46 LBA:46464646 DAT:4346 SC:46

IS:0; stTo:2; drc:1; F:1/5367 -25014ms

From Dumpcheck also we could see that there is a trace back for the card-1, which

is not normal.

2> os excDumpCheck

Software error PRESENT on Card 0 (Standby CP)

Traceback and Software error PRESENT on Card 1 (Active CP)

Software error PRESENT on Card 2 (FP)











After CP Outage restored in RNC, the locked PMP associations alarms raised

again... as below...

23016 34980/1 34985/41 ; 2015-08-15 04:55:06.73

SET major communications 70790202


AVAIL: PROC: CNTRL:


Id: 9

Com: Pmp is down

Int: 0/1/2/24425; m3uaPmp.cc; 901; RID37300E1953

23016 34980/1 34985/42 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: A

Com: Pmp is down


23016 34980/1 34985/56 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: B

Com: Pmp is down


23016 34980/1 34985/67 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: C

Com: Pmp is down


23016 34980/1 34985/68 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: D

Com: Pmp is down


23016 34980/1 34985/69 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: E

Com: Pmp is down


23016 34980/1 34985/70 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: F

Com: Pmp is down


23016 34980/1 34985/71 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: 10

Com: Pmp is down


23016 34980/1 34982/7 ; 2015-08-15 04:55:06.73

SET critical communications 70790200


AVAIL: PROC: CNTRL:


Id: 11

Com: DestSp is inaccessible

Int: 0/1/2/24425; m3uaDestSp.cc; 458; RID37300E1953

23016 34980/1 34982/14 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: 12

Com: DestSp is inaccessible

Int: 0/1/2/24425; m3uaDestSp.cc; 458; RID37300E1953

23016 34980/1 34986/7 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: 13

Com: Pme is down

Int: 0/1/2/24425; m3uaPme.cc; 631; RID37300E1953

23016 34980/1 34986/14 ; 2015-08-15 04:55:06.73



AVAIL: PROC: CNTRL:


Id: 14

Com: Pme is down

Int: 0/1/2/24425; m3uaPme.cc; 631; RID37300E1953

From the Latest RNC health check I could see that again card-1 is active ...as

below..

###################################################################

### RNC235 STATE CHECKER --- 15/08/2015 at 14:00:17

### -------------------------------------------------

--- RNC Cards list ---

--> Passport's card list : | CP : 2 | PSFP: 12 | OC3: 0 | 4pGe: 2 |

--> RNC overload status is ok

CARD TYPE FSM STATE VISIBILITY LP INSTANCE CPU Util %

---- ---------- ----------------- ---------- -- -------- ----------

0 CPeE RunningAsStandby OnLine 0 Standby 3

1 CPeE RunningAsActive OnLine 0 Active 8

From the card uptime also we can conclude that issue started with CP-1 TRAP disk

error issue due to which FDDCell unlock command failed and was incomplete in WMS

till you reset the process in WMS for recovering it. So request you to provide the

job history logs to confirm the same.

1> d sh ca/* carduptime

Shelf Card/*

+====+------------------------------------------------------

|Card| cardUpTime

| | days hours minute second usecon

+====+------------------------------------------------------

| 0| 0 7 40 48 674428 -' Need to confirm from logs whether it is due to CP Switchover or not?

| 1| 0 12 42 52 315787 -' reset happened due to TRAP issue.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212


Time Tracking Entry Added - IR01-TSA3

13. 18-Aug-2015 12:16 pthota


requested few more logs and details. waiting for those details.


From: MERAD, ABDESSAMAD (ABDESSAMAD)


To: THOTA, Praveen-K (Praveen-K)** CTR **; IBRAHIM ABD EL NABY, Karim (Karim)**

CTR **


(YASSER)** CTR **; ABDEL-HALIM, SAYED (SAYED); EL-MIDANY, AHMED (AHMED);

INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **; INDIA-WCDMA-WMS-NPO-BE



Importance: High

Please summarize the required actions

thanks

Best Regards



To: MERAD, ABDESSAMAD (ABDESSAMAD); THOTA, Praveen-K (Praveen-K)** CTR **



INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **; INDIA-WCDMA-WMS-NPO-BE



Hello Praveen

Thanks for your detailed analysis, please help to the RCAs next steps also,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798






To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; MERAD, ABDESSAMAD (ABDESSAMAD)



INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **; INDIA-WCDMA-WMS-NPO-BE;

CHOWDARY, Ajay (Ajay)** CTR **



++

Hi Karim/Merad,

Below I am summarizing the issue...

As indicated previously the LOS issue on RNC 235 and NodeBs under it, is not only

with RNC 235. I think there is some external issue, due to which all RNCs are

having LOS at the same time.

From HFB the issue is observed on all 4 legacy RNCs on 14-08-2015 at 17:06 &

15-08-2015 at 09:35 and LOS observed for all 6 RNCs(including WCE RNCs) on

15-08-2015 at 09:08 has to be investigated separately with the help of WMS support

team.

Adding Mr. Ajay who is investigating from WMS point of view. As requested by him,

please provide below logs from WMS for further investigation of the issue.

/opt/nortel/logs/utran

o cp_utran-20000.log

o cp_utran-20000.tra

o WOM_ADAPTER-20002.log

o WOM_ADAPTER-20002.tra

o WOM_COMM_SEPE_RNC-20010.tra

o WOM_COMM_SEPE_RNC-20010.log

From the RNC Point of view LOS observed on 15-08-2015 at 04:55 was due to Shelf

Card/1 TRAP with disk Error issue. From the card installation to till date this

card crashed for two times, So it is advised to replace this card in Maintenance

window to avoid further issues.

For further investigation of the issues and LOS observed for RNC 235 between 09:00

AM to 10:00 AM on 15-08-2015. I request below logs, thanks to provide the same at

the earliest.

Note : previously provided RNC logs are having details till 2015-08-15 06:13:09.61

only.

1) RNC Log collection full for this RNC ' Please collect it again and upload

it to the server.

2) Also request you to provide HFB for this RNC for 15th August 2015. (if not

please upload the HFB file from WMS stability folder for August 15th date to the

server).

3) /opt/nortel/data/cmXML/CMSessions/YYYYMMDD/Online+Commands/* -' All files

from this path for 15th August 2015.

4) /opt/nortel/logs/wtk/JobHistory-00114* -' All Job History files of August

15th 2015.

5) Screenshot of the unlock command struck in WMS, if not available you can

take it from WMS commands history.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212

15. 19-Aug-2015 13:15 pthota


Issue was caused due to CP card crash from slot-1, recommended to replace the

board in maintenance window.



Sent: Tuesday, August 18, 2015 2:41 PM

To: THOTA, Praveen-K (Praveen-K)** CTR **; CHOWDARY, Ajay (Ajay)** CTR **



INDIA-WCDMA-UTRAN-BE; ARORA, Gaurav (Gaurav)** CTR **; MERAD, ABDESSAMAD

(ABDESSAMAD); INDIA-WCDMA-WMS-NPO-BE



Hello Praveen,Ajay,

Thanks for your feedback, Ramses FTP is available so please download the required

logs directly from it,

For the RNClog collection may I ask why it required again , as the one taken

during the action not help you, so why requested again now,

Please note the lOS happened as we did WMS stop/start in the same mentioned all

times as we have another issue with other.

Please provide final/clear reason for the issue so we can provide to the customer

clear reason for this issue, or you can confirm that the CP card is the main

RCA?!.

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798




To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; CHOWDARY, Ajay (Ajay)** CTR **







Hi Karim,

I think you missed to read the reason from below for asking RNC Log collection

again, anyway please find it again...

Ramses FTP is available so please download the required logs directly from it,

No problem, we will collect the ftp possible logs from WMS directly. Thanks

For the RNClog collection may I ask why it required again , as the one taken

during the action not help you, so why requested again now,

From the provided RNC 235 RNC Log collection, which was collected at 08:15 AM of

August 15th, I could see the alarms till 2015-08-15 06:13:09.61 only.

So request you to run the RNC Log collection and let me know the file to be

collected, but please note it will take longer time to download it from Ramses

directly.

Please provide below logs, as it cannot be collected through Ramses....

1) Screenshot of the unlock command struck in WMS, if not available you can

take it from WMS commands history.

Please note the lOS happened as we did WMS stop/start in the same mentioned all

times as we have another issue with other. -' I didn't understand, could you

please explain in detailed....

Please provide final/clear reason for the issue so we can provide to the customer

clear reason for this issue, or you can confirm that the CP card is the main

RCA?!.

After checking the below requested logs, we will try to provide you the clear

reason for the LOS....!!!

But I just want to inform you again that the LOS issue not happened alone to RNC

235 at that time as per below LOS alarms of 15th August 2015, do you agree?

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212



To: THOTA, Praveen-K (Praveen-K)** CTR **; CHOWDARY, Ajay (Ajay)** CTR **







Hello Praveen

OK, thanks , please as I mentioned to you this loss of supervision is due to

stop/start of the WMS, that's why you see it on all the RNCs,

I'll collect the rnclog again and share with you,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798








(ABDESSAMAD); INDIA-WCDMA-WMS-NPO-BE; CHOWDARY, Ajay (Ajay)** CTR **



Hi Karim,

The alarms observed on 15-08-2015 at 09:08 AM for all RNCs including WCE RNCs may

be due to the processes stop and start in WMS -' Could you please confirm at what

time the WMS processes stop/start done and for how many times?

The LOS observed for four RNCs only (235,231,232 and 239) on 14-08-2014 at 17:06

PM and 15-08-2015 at 09:35 AM also due to WMS stop/start activity? Please confirm.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212










Hello Praveen

Confirmed,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798



Sent: Wednesday, August 19, 2015 9:38 AM








Hi Karim,

I have gone through the Latest RNC Log collection of this RNC 235 and also checked

the RNC Log collection of RNC 239 and nothing found abnormal.

In RNC235 I didn't find anything abnormal during or before CP switchover time.

From the logs analysis the only issue found was CP-1 card crash at around 5:00 AM.

Since there is no other abnormal behavior observed, This issue happened in this

RNC could be due to CP-1 card crash only and consequently it has not recovered

automatically when one process stop/start performed from WMS and also was not

recovered like other RNCs when WMS all processes stop/start done. Finally it

recovered with CP-Switchover action.

As already communicated below, For the Shelf Card/1 card crash issue. We could see

this card crashed for two times as below from the card installation to till date,

So it is advised to replace this card at the earliest possible time in Maintenance

window to avoid further issues.

Please let me know if you need any further analysis for the same.

Old Crash info :

Unrecoverable error at line 2047 in file sfsFileSys.cc during interrupt 236

(0x000000ec), currenttask "tCardReset" (0x1c015598)

Standby CP is reset by active CP due to a problem detected by SFS.

Error Info: (0.19)

*** An exception has occurred at interrupt level ***

Offending interrupt 236 (0xec) was near address 0x019cf91c, current task

"tCardReset" (0x1c015598)

(errno = 0x029c0000)

Card Info: (0.25)

PEC Code : NTPN08AG-10

Serial number : NNTMPJ00WFK2

PM Rev Code : NTPN2632-12

Reset Info: (0.17)

Reboot Reason value is 400 [hardware watchdog reset]

Last boot type: (cold autoboot)

Latest Crash :



--- Reason ---


[sfs:disk test]

Error Info: (0.18)




Card Info: (0.26)

PEC Code : NTPN08AG-10

Serial number : NNTMPJ00WFK2

PM Rev Code : NTPN2632-12

Reset Info: (0.17)

No significant reboot reason available

Last boot type: (warm autoboot)

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212


Sent: Wednesday, August 19, 2015 12:03 PM








Hello Praveen

Thanks for your detailed invistagation,

So finally that issue with CP1 and happened due to the crash of it,

Please what about the below errors we shared at the beginning of the AR? Ar there

any issue or not?

2> os excDumpCheck

Software error PRESENT on Card 0 (Standby CP)

Traceback and Software error PRESENT on Card 1 (Active CP)












Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798













Hi Karim,

The below os excDumpCheck command display the information related to the trace

back and software errors which were present on these cards previously.

From RNC Log collection I could see the crash report for card-1 only in this RNC,

I didn't find it for any other cards. All the other cards which are having

software errors could be recoverable hence there is no crash reports for them. If

any card had unrecoverable error one crash report will be created for that card

for each error.

If you want to diagnose more on these software errors of other cards, you may open

a new AR for this. Thanks for understanding.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212




18. 19-Aug-2015 18:32 ALCATEL-LUCENT PROPRIETARY nchatter


19. 20-Aug-2015 08:56 pthota







To: THOTA, Praveen-K (Praveen-K)** CTR **; IBRAHIM ABD EL NABY, Karim (Karim)**

CTR **







Hello Praveen

Kindly note that we will plan for replacement for CP1 shortly

Please share the latest update MoP for current SW.

Thanks for your support,

Karim

+97156 3544568





Sent: Thursday, August 20, 2015 11:05 AM








Hi Karim,

PFA Maintenance guide for RNC 9370. Please refer section "Replacement of a CP in a

two-CP node" from page No : 108 in Maintenance activities.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212





24. 24-Aug-2015 09:27 ALCATEL-LUCENT PROPRIETARY gauravarora


25. 24-Aug-2015 09:28 ALCATEL-LUCENT PROPRIETARY gauravarora


26. 24-Aug-2015 14:09 pthota



board in maintenance window. Closure confirmation received to close this AR.











Hi Karim,

If no other query on this issue, can we proceed to close this AR? Thanks to

confirm.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212










Hello Praveen

Thanks you can close the AR, and please update the final RCA on cares if not

updated,

Thanks,

BR,

Karim Ibrahim



M: +971(0)563544568| ONNET: 22341798













Thanks Karim for closure confirmation. Sure, I will update the AR with RCA details

before closing it in cares.

Thanks and Regards,

Praveen Kumar Thota

Mob: +91 96899 18050

Desk: +91 80 66390212


Add to log

Resolution



End of Assistance Request 1-5877056

New AR: View AR | AR without Proprietary | OLCS

History | States | Assignment | Timetracking

View Attachments | Add Attachment

AR Text Search | More | CARES Home

AR5

Documents

Transcript of AR5