Fault Management Automation, FMX (2G - 3G)

45
Fault Management Automation FMX

description

Operation

Transcript of Fault Management Automation, FMX (2G - 3G)

Page 1: Fault Management Automation, FMX (2G - 3G)

Slide titleIn CAPITALS

50 pt

Slide subtitle 32 pt

Fault Management AutomationFMX

Page 2: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-292

Presentation agenda

Overview Challenges in Supervision with FMX Experience from other markets FMX Services and products Alarm log analysis GSM/CN Network Alarm log analysis WCDMA Network Rule Development process FMX Know-how Complete FMX solutions

Page 3: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-293

Overview

OSS-RC

FM(FMX) PM CM SM

Telecom Network

NE

NE

NE

NE

NE

NE NE

NE

NENE

2G3G

CORE

IMS

TeMIP/Netcool (NMX)

NMS system

Trouble Ticketing

Page 4: Fault Management Automation, FMX (2G - 3G)

Slide titleIn CAPITALS

50 pt

Slide subtitle 32 pt

Do you have a clear view?

Page 5: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-295

Challenges in supervision

Poor visibility in supervision – Alarm storms problem– Frequent alarms– Several alarms indicate the same problem

CCITT7 SIGNALLING LINK FAILURE

DIGITAL PATH FAULT SUPERVISION

DIGITAL PATH QUALITY SUPERVISION

CELL LOGICAL CHANNEL AVAILABILITYSUPERVISIONRADIO X-CEIVER ADMINISTRATIONMANAGED OBJECT FAULT

Page 6: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-296

Challenges in supervision

Poor visibility in supervision – Alarm storms problem– Frequent alarms– Several alarms indicate the same problem

Network growth with same or reduced staff OPEX reduction Improvement of network availability

Page 7: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-297

Way forward

Increase visibility

Increase visibility

Show problem

Show problem

Automatic diagnostics &

repair

Automatic diagnostics &

repair

OPEX reductionHandle network growth

Network availability improved

OPEX reductionHandle network growth

Network availability improved

Page 8: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-298

Automatic diagnostics &

repair

Automatic diagnostics &

repair

Get a clear view with FMX

Increase visibility

Increase visibility

Show problem

Show problem

Prioritize the alarmsFilter short lived alarms

Alarm enrichment

Prioritize the alarmsFilter short lived alarms

Alarm enrichment

Execute repair sequence Alarm enrichment (db´s,mml)

Create Trouble Tickets

Execute repair sequence Alarm enrichment (db´s,mml)

Create Trouble Tickets

Hide secondary alarmsShow problem, not alarms (IUB down)

Create frequent alarms

Hide secondary alarmsShow problem, not alarms (IUB down)

Create frequent alarms

Page 9: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-299

Scenario 1 – too many alarms

3G Network with 40.000 alarms/day

Small operational staff

Hard to distinguish between important problems and noise

Experience from other markets

Page 10: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2910

Scenario 1 - FMX Solution

FMX Filter and correlation rules

Process 86% of all alarms

Filter out 67% of total alarm flow

Remaining 13.200 Alarms/Day of 40.000->Time saved 1-3 hours / day & operator

Increase visibility

Increase visibility

Show problem

Show problem

Experience from other markets

Page 11: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2911

Scenario 2 – automatic information gathering

2G Network with 20.000 alarms/day

2800 Radio alarms from BTS(RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT)

40% Critical/A1

Operators has to send MML command to AXE to retrieve fault codes

Experience from other markets

Page 12: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2912

Scenario 2 – FMX Solution

FMX send command to the switches for all Critical alarms

Fault codes are translated into text and presented in the alarm text

FMX sends 1120 commands/day

->Time saved >9 hours / day

Increase visibility

Increase visibility

Automatic diagnostics &

repair

Automatic diagnostics &

repair

Experience from other markets

Page 13: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2913

Scenario 3 – Automatic actions

2G Network with 8.500 alarms/day

8% Radio alarms from BTS(RADIO X_CEIVER ADMINISTRATION MANAGED OBJECT FAULT)

24% Time slot fault (TS SYNC FAULT)

Usually block and de-block of the time slot solves the problem

Experience from other markets

Page 14: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2914

Scenario 3 – FMX Solution

FMX send commands to the switches for all time slot alarms(RXBLI and RXBLE)

FMX block and de-block 338 timeslots/day

->Time saved 3-4 hours/day

Automatic diagnostics &

repair

Automatic diagnostics &

repair

Increased scope:

Info number percentCELL LOGICAL 2507Filtered 1120 45%Solved 1167 47%Alarm updated 220 8%

-> Increased scope >50 hours/day

Increased scope:

Info number percentCELL LOGICAL 2507Filtered 1120 45%Solved 1167 47%Alarm updated 220 8%

-> Increased scope >50 hours/day

Experience from other markets

Page 15: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2915

Show me the money

Assumption: – 2G network for large operator– Automation of the network using FMX

Result:– Unavailability reduced ( 70%) – 5.400 man hours saved / year– Solution management contracts

Experience from other markets

Page 16: Fault Management Automation, FMX (2G - 3G)

Slide titleIn CAPITALS

50 pt

Slide subtitle 32 pt

FMX Services and products

Complete FMX solutions and FMX know-how

FMX Service description

Page 17: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2917

FMX Service options flow

Alarm Log

Analysis

FMX Rules

on Demand

FMX Surveillance expert integration and configuration

FMX Starter package

(CNX, WRX)

Solution Support & LCM

SolutionManagement

SolutionIntegration

Solution Analysis

Page 18: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2918

Alarm log analysisSummary for GSM/CN Network

November 2008, 16 and 21 days (madmas2o & bnamas2o)

2.605.945 alarms received:– 1.293.063 alarms on Madrid-OSS (madmas2o)– 1.312.882 alarms on Barcelona-OSS (bnamas2o)

71.667 alarms per day:– 80.816 alarms per day on Madrid-OSS (madmas2o)– 62.518 alarms per day on Barcelona-OSS (bnamas2o)

Results for GSM/CN Network

Page 19: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2919

Alarm log analysisTop 10 frequent alarms

Results for GSM/CN Network

The 10 most frequent alarms mean the 67% of total alarms

SpecificProblemTotal number of

alarms

Percent of total alarm

flow

RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT 767287 29%

RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT 250324 10%

TEST SYSTEM ACTIVATED 164813 6%

CCITT7 SIGNALLING LINK FAILURE 139105 5%

DIGITAL PATH FAULT SUPERVISION 109838 4%

DIGITAL PATH QUALITY SUPERVISION 96000 4%

EXTERNAL ALARM 83782 3%

FSC-SS7Manager/B/2 : MTPL-3: Link Out Of Service 83113 3%

DIGITAL PATH UNAVAILABLE STATE FAULT 27716 1%

HEARTBEAT FAILURE 26585 1%

TOTAL 1748563 67%

Page 20: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2920

Alarm log analysisTop 10 short-live* alarms

Results for GSM/CN Network

The 51% of alarms are cleared in less than 5 minutes

*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )

SpecificProblemTotal

number of alarms

Total number of shortlived

alarms

Shortlived percentage

Shortlived percentage of all alarms

RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT 767287 596415 78% 23%

TEST SYSTEM ACTIVATED 164813 162903 99% 6%

CCITT7 SIGNALLING LINK FAILURE 139105 131225 94% 5%

RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT 250324 111774 45% 4%

DIGITAL PATH FAULT SUPERVISION 109838 98515 90% 4%

FSC-SS7Manager/B/2 : MTPL-3: Link Out Of Service 83113 83002 100% 3%

EXTERNAL ALARM 83782 72417 86% 3%

DIGITAL PATH QUALITY SUPERVISION 96000 24616 26% 1%

HEARTBEAT FAILURE 26585 23761 89% 1%

DIGITAL PATH UNAVAILABLE STATE FAULT 27716 20935 76% 1%

TOTAL 1748563 1325563 51%

Page 21: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2921

Alarm log analysisSuggested FMX Rules (I)

Results for GSM/CN Network

Blacklisting of nodes Filter out all the alarms from the nodes under maintenance in the

blacklist

RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT Single slogan presented (MAINS FAIL, DOOR OPEN and SITE

ON BATTERY), Filters short-lived, Filters unimportant alarms and Frequent alarms created.

Short-lived filter will reduce the alarm flow with on average ~23%

Page 22: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2922

Alarm log analysisSuggested FMX Rules (II)

Results for GSM/CN Network

RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT Filters short-lived and Create FREQUENT alarm Automatic block/deblock on TS Presents DIP & STATE if OML FAULT Gets affected CELL to enable correlation with the service alarm CELL

LOGICAL CHANNELS AVAILABILITY SUPERVISION RXMFP is sent, the fault codes are translated to text and presented

with the alarm. An alarm reduction of ~4% filtering short-lived alarms only

DIGITAL PATH FAULT SUPERVISION Filter short-lived, Frequent alarms created Correlation with the DIGITAL PATH UNAVAILABLE STATE FAULT

alarm so that if both alarms appear on the same MO and DIP only one of them are shown.

An alarm reduction of ~4% filtering short-lived alarms only

Page 23: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2923

Alarm log analysisSuggested FMX Rules (III)

Results for GSM/CN Network

DIGITAL PATH QUALITY SUPERVISION Will be automatically reset by the rule (DTQSR), i.e.maximum 3 times

on the same DIP within 24 hours If alarm comes 3 days in a row, same DIP, a FREQUENT error

message is created

CCITT7 SIGNALLING LINK FAILURE Filter short-lived, Frequent alarms created Not short-lived can be analyzed to find the related DIP, DIP-STATE and

if applicable the route-name (R). An alarm reduction of ~5% filtering short-lived alarms only

EXTERNAL ALARM Filters short-lived with non important alarms slogans Frequent alarms created for non important slogans

Page 24: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2924

Alarm log analysisSuggested FMX Rules (IV)

Results for GSM/CN Network

TEST SYSTEM ACTIVATED Unless the operators keep track of whether a test system is active, a

reduction of the total alarm flow of about ~6%

FSC-SS7Manager/B/2: MTPL-3: Link Out Of Service Filter short-lived, Since most of these alarms are short-lived (~99%) An alarm reduction of ~3% filtering short-lived alarms only

Filtering short-lived alarms only 51% of total alarm flow can be cleared

Page 25: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2925

Alarm log analysisSummary for WCDMA Network

December 2008, 18 days (Coruña rcmser01 & Leganés rcmser02)

1.207.633 alarms received:– 774.387 alarms on Coruña-OSS (rcmser01)– 433.246 alarms on Leganés-OSS (rcmser02)

67.090 alarms per day:– 43.021 alarms per day on Coruña-OSS (rcmser01)– 24.069 alarms per day on Leganés-OSS (rcmser02)

Results for WCDMA Network

Page 26: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2926

Alarm log analysisTop 20 frequent alarms Coruña-OSS (rcmser01)

Results for WCDMA Network

Specific Problem Object Type Number %

Ip over ATM Link Supervision Recovered URAN 320988 41%

Ip over ATM Link Supervision Failed URAN 320734 41%

AntennaBranch_AntennaSystemProblemInBranchA URAN 12786 2%

AlmDevice_ExternalAlarm URAN 10567 1%

DbccDevice_GammaDownlinkFailure URAN 9625 1%

IMA Link Transmit Unusable at Far End URAN 8326 1%

IMA Group Insufficient Links at Far End URAN 7725 1%

Heartbeat Failure URAN 7666 1%

NodeSynchTp_ConnectionLost URAN 7570 1%

UtranCell_NbapMessageFailure URAN 7268 1%

UtranCell_ServiceUnavailable URAN 5874 1%

IMA Link Reception Unusable at Far End URAN 5843 1%

IMA Group Insufficient Links URAN 4906 1%

Remote Defect Indication on IMA Link URAN 4372 1%

NssSynchronization_SynchRefChanged URAN 4239 1%

NssSynchronization_SystemClockStatusChanged URAN 4186 1%

PDH Alarm Indication Signal URAN 3364 0%

IMA Group Configuration Aborted URAN 2873 0%

RruDeviceGroup_FanFailure URAN 1946 0%

NbapDedicated_RncRbsControlLinkDown URAN 1612 0%

Total   752470 97%

Page 27: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2927

Alarm log analysisTop 20 frequent alarms Leganés-OSS (rcmser02)

Results for WCDMA Network

Specific Problem Object Type Number %

Ip over ATM Link Supervision Recovered URAN 63284 15%

Ip over ATM Link Supervision Failed URAN 62801 14%

AlmDevice_ExternalAlarm URAN 60214 14%

UtranCell_ComMeasFailCongCon URAN 31318 7%

IMA Link Reception Unusable at Far End URAN 24278 6%

IMA Link Transmit Unusable at Far End URAN 21346 5%

Heartbeat Failure URAN 16769 4%

Remote Defect Indication on IMA Link URAN 14146 3%

AntennaBranch_AntennaSystemProblemInBranchA URAN 11450 3%

NssSynchronization_SynchRefChanged URAN 9292 2%

IMA Group Insufficient Links URAN 8921 2%

NssSynchronization_SystemClockStatusChanged URAN 8309 2%

UtranCell_ServiceUnavailable URAN 7441 2%

NodeSynchTp_ConnectionLost URAN 7187 2%

UtranCell_NbapMessageFailure URAN 7141 2%

PDH Alarm Indication Signal URAN 6041 1%

DbccDevice_GammaDownlinkFailure URAN 5371 1%

IMA Group Insufficient Links at Far End URAN 5231 1%

OpticalInterfaceLink_OpticalInterfaceLinkFailure URAN 4803 1%

NodeSynch_Phase_Difference_Measurement_Failed URAN 4549 1%

Total   379892 88%

Page 28: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2928

Alarm log analysisTop 10 short-live* alarms Coruña-OSS (rcmser01)

Results for WCDMA Network

*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )

SpecificProblem Object TypeTotal

number

Total number shortlived with duration less

than 5 minutes.

Percentage of

shortlived

Percent of Shortlived

of total

DbccDevice_GammaDownlinkFailure URAN 9625 8834 92% 1%

AntennaBranch_AntennaSystemProblemInBranchA URAN 12786 8801 69% 1%

AlmDevice_ExternalAlarm URAN 10567 8117 77% 1%

Ip over ATM Link Supervision Recovered URAN 320988 8095 3% 1%

Ip over ATM Link Supervision Failed URAN 320734 7851 2% 1%

IMA Link Transmit Unusable at Far End URAN 8326 7828 94% 1%

IMA Group Insufficient Links at Far End URAN 7725 7492 97% 1%

NodeSynchTp_ConnectionLost URAN 7570 6935 92% 1%

UtranCell_NbapMessageFailure URAN 7268 6878 95% 1%

Heartbeat Failure URAN 7666 5440 71% 1%

Total   713255 76271 10% 10%

Page 29: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2929

Alarm log analysisTop 10 short-live* alarms Leganés-OSS (rcmser02)

Results for WCDMA Network

*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )

SpecificProblemObject Type

Total number

Total number shortlived

with duration less than 5 minutes.

Percentage of

shortlived

Percent of Shortlived

of total

AlmDevice_ExternalAlarm URAN 60214 34617 57% 8%

UtranCell_ComMeasFailCongCon URAN 31318 30974 99% 7%

Ip over ATM Link Supervision Recovered URAN 63284 24004 38% 6%

Ip over ATM Link Supervision Failed URAN 62801 23920 38% 6%

IMA Link Reception Unusable at Far End URAN 24278 13499 56% 3%

IMA Link Transmit Unusable at Far End URAN 21346 12108 57% 3%

Remote Defect Indication on IMA Link URAN 14146 9212 65% 2%

Heartbeat Failure URAN 16769 8259 49% 2%

UtranCell_NbapMessageFailure URAN 7141 6307 88% 1%

AntennaBranch_AntennaSystemProblemInBranchA URAN 11450 6108 53% 1%

Total   312747 169008 54% 39%

Page 30: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2930

Alarm log analysisSuggested FMX Rules (I)

Results for WCDMA Network

Module Handled Alarms (Italic = FMX created) WRX_IUB_disturbances UtranCell_ServiceUnavailable IUB down FREQUENT IUB disturbances Fach_InternalResourceUnavailable Pch_InternalResourceUnavailable Rach_InternalResourceUnavailable NbapDedicated_RncRbsControlLinkDown FREQUENT NbapDedicated_RncRbsControlLinkDown WRX_NodeB_disturbances UtranCell_NbapMessageFailure UtranCell_RbsFailure NodeB down FREQUENT NodeB disturbances WRX_AuxPlugInUnit_PiuConnectionLost AuxPlugInUnit_PiuConnectionLost FREQUENT AuxPlugInUnit_PiuConnectionLost WRX_NodeSynchTp_ConnectionLost NodeSynchTp_ConnectionLost FREQUENT NodeSynchTp_ConnectionLost WRX_PlugInUnitFault Plug-In Unit Fault FREQUENT Plug-In Unit Fault PlugInUnit_BoardSwitchCoreFault FREQUENT PlugInUnit_BoardSwitchCoreFault SwitchModule_SwitchPlaneAFault

Page 31: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2931

Alarm log analysisSuggested FMX Rules (II)

Results for WCDMA Network

Module Handled Alarms (Italic = FMX created) WRX_UtranCell_ComMeasFailCongCon UtranCell_ComMeasFailCongCon FREQUENT UtranCell_ComMeasFailCongCon WRX_Utran_External_Alarm AlmDevice_ExternalAlarm FREQUENT AlmDevice_ExternalAlarm WRX_VCeteAlarmIndicationSignal VC ete Alarm Indication Signal FREQUENT VC ete Alarm Indication Signal WRX_CchFrameSynch_TimingAdjCtrlFrame CchFrameSynch_TimingAdjCtrlFrame FREQUENT CchFrameSynch_TimingAdjCtrlFrame WRX_Clu_LossOfMains Clu_LossOfMains FREQUENT Clu_LossOfMains Blocking Filters NbapCommon_RncRbsControlLinkDown NbapDedicated_RncRbsControlLinkLossOfRedundancy NbapCommon_RncRbsControlLinkLossOfRedundancy NbapCommon_SuccessfulReconfigurationAtFailure NbapCommon_SuccessfulReconfigurationOrdered NbapDedicated_SuccessfulReconfigurationAtFailure NbapDedicated_SuccessfulReconfigurationOrdered SYSTEM_CLOCK_IN_HOLDOVER_MODE System Clock in Holdover Mode E1Ttp_NOCRC4MFA NssSynchronization_SystemClockStatusChanged NodeBFunction_O&MapplicationRestarted SlotMo New PIU detected, matches the defined configuration AscDeviceGroup_SuccessfulRecoveryActionPerformedBoardRestart

Page 32: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2932

Alarm log analysisEstimated Alarm Reduction Coruña-OSS (I)

Results for WCDMA Network

Specific Problem Number % Type of filterEst. % alarm

reduction Filtered

Ip over ATM Link Supervision Recovered 320988 41%Filter with WRX Block 100% 320988

Ip over ATM Link Supervision Failed 320734 41%Filter with WRX Block 100% 320734

AntennaBranch_AntennaSystemProblemInBranchA 12786 2%     0

AlmDevice_ExternalAlarm 10567 1%correlation, short-lived filter, frequent 77% 8137

DbccDevice_GammaDownlinkFailure 9625 1%     0

IMA Link Transmit Unusable at Far End 8326 1%     0

IMA Group Insufficient Links at Far End 7725 1%     0

Heartbeat Failure 7666 1%     0

NodeSynchTp_ConnectionLost 7570 1% short-lived filter, frequent  92% 6964

UtranCell_NbapMessageFailure 7268 1%correlation, short-lived filter, frequent 95% 6905

UtranCell_ServiceUnavailable 5874 1%correlation, short-lived filter, frequent 49% 2878

IMA Link Reception Unusable at Far End 5843 1%     0

IMA Group Insufficient Links 4906 1%     0

Remote Defect Indication on IMA Link 4372 1%     0

NssSynchronization_SynchRefChanged 4239 1%Filter with WRX Block 100% 4239

Alarm is candidate for Rules on Demand package

Alarm is processed by WRX package

No defined FMX package for alarm

Page 33: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2933

Alarm log analysisEstimated Alarm Reduction Coruña-OSS (II)

Results for WCDMA Network

Specific Problem Number % Type of filterEst. % alarm

reduction Filtered

NssSynchronization_SystemClockStatusChanged 4186 1% Auto-ack/ FMX filter 100% 4186

PDH Alarm Indication Signal 3364 0%     0

IMA Group Configuration Aborted 2873 0%     0

RruDeviceGroup_FanFailure 1946 0%     0

NbapDedicated_RncRbsControlLinkDown 1612 0% Auto-ack/ FMX filter 100% 1612

AuxPlugInUnit_SuccessfulRecoveryActionPerformed 1575 0%     0

OpticalInterfaceLink_OpticalInterfaceLinkFailure 1219 0%     0

PacketDataRouter_CnNotRespondingToGTPEcho 1153 0%     0

E1Ttp_NOCRC4MFA 1001 0% Auto-ack/FMX-filter  100% 1001

PDH Remote Defect Indication 938 0%     0

Rach_InternalResourceUnavailable 917 0%correlation, short-lived

filter, frequent 97% 889

Fach_InternalResourceUnavailable 836 0%correlation, short-lived

filter, frequent 98% 819

Pch_InternalResourceUnavailable 825 0%correlation, short-lived

filter, frequent 98% 809

Loss of Cell Delineation 822 0%     0

PDH Loss of Signal 812 0%     0

Total 762568 98%    87% 680161

Alarm is candidate for Rules on Demand package

Alarm is processed by WRX package

No defined FMX package for alarm

Page 34: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2934

Alarm log analysisEstimated Alarm Reduction Leganés-OSS (I)

Results for WCDMA NetworkAlarm is candidate for Rules on Demand package

Alarm is processed by WRX package

No defined FMX package for alarm

Specific Problem Number % Type of filterEst. % alarm

reduction Filtered

Ip over ATM Link Supervision Recovered 63284 15%Filter with WRX Block 100% 63284

Ip over ATM Link Supervision Failed 62801 14%Filter with WRX Block 100% 62801

AlmDevice_ExternalAlarm 60214 14%correlation, short-lived filter, frequent 57% 34322

UtranCell_ComMeasFailCongCon 31318 7%correlation, short-lived filter, frequent 99% 31005

IMA Link Reception Unusable at Far End 24278 6%     0

IMA Link Transmit Unusable at Far End 21346 5%     0

Heartbeat Failure 16769 4%     0

Remote Defect Indication on IMA Link 14146 3%     0

AntennaBranch_AntennaSystemProblemInBranchA 11450 3%     0

NssSynchronization_SynchRefChanged 9292 2%Filter with WRX Block 100% 9292

IMA Group Insufficient Links 8921 2%     0

NssSynchronization_SystemClockStatusChanged 8309 2%Auto-ack/ FMX filter 100% 8309

UtranCell_ServiceUnavailable 7441 2%correlation, short-lived filter, frequent 38% 2828

NodeSynchTp_ConnectionLost 7187 2% short-lived filter, frequent  58% 4168

Page 35: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2935

Alarm log analysisEstimated Alarm Reduction Leganés-OSS (II)

Results for WCDMA NetworkAlarm is candidate for Rules on Demand package

Alarm is processed by WRX package

No defined FMX package for alarm

Specific Problem Number % Type of filterEst. % alarm

reduction Filtered

UtranCell_NbapMessageFailure 7141 2%correlation, short-lived filter, frequent 88% 6284

PDH Alarm Indication Signal 6041 1%     0

DbccDevice_GammaDownlinkFailure 5371 1%     0

IMA Group Insufficient Links at Far End 5231 1%     0

OpticalInterfaceLink_OpticalInterfaceLinkFailure 4803 1%     0

NodeSynch_Phase_Difference_Measurement_Failed 4549 1%     0

NodeBFunction_NodeRestarted 4526 1%     0

PDH Remote Defect Indication 4318 1%     0

IMA Group Configuration Aborted 4069 1%     0

UtranCell_ComMeasFailAdmCon 2793 1%     0

Pch_InternalResourceUnavailable 2712 1%correlation, short-lived filter, frequent 87% 2359

Fach_InternalResourceUnavailable 2646 1%correlation, short-lived filter, frequent 86% 2276

Rach_InternalResourceUnavailable 2611 1%correlation, short-lived filter, frequent 85% 2219

NodeBFunction_MpTrafficApplicationRestarted 2410 1%     0

IMA Link Transmission Misconnected 2322 1%     0

ObifDeviceGroup_PreloadFailed 2305 1%     0

Total 410604 95%    53% 229147

Page 36: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2936

Alarm log analysisEstimated Alarm Reduction

Results for WCDMA Network

With the WRX rules package, an estimated 87% and 53% of total alarm flow reduction may be achieved on

Coruña-OSS and Leganés-OSS respectively

The reduction level in the table is based on 5 minutes filtering for short lived alarms

Page 37: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2937

Alarm log analysisRules on Demand

Results for WCDMA Network

By using simple short-lived filtering, a further 4% and 11% of total alarm flow reduction may be achieved on Coruña-OSS and Leganés-OSS .

The focus would be on the IMA alarms group:– IMA Link Transmit Unusable at Far End– IMA Group Insufficient Links at Far End– IMA Link Reception Unusable at Far End– IMA Group Insufficient Links– Remote Defect Indication on IMA Link– IMA Group Configuration Aborted

Page 38: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2938

Rule Development ProcessWhat can we do with FMX?

“If you can write it down, you can

implement it with FMX”

“If you can write it down, you can

implement it with FMX”

!

Page 39: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2940

Rule Development ProcessAvailable features used

Rules are grouped in independent modules All controllable parameters are set in a text file, can be

updated in run-time Rule statistics available

Page 40: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2941

Blocking Filters Threshold filters

Alarm-based correlation

Duration filters

Topology-based

correlation

Effort to implement reduction

Rule Development ProcessEffect vs Effort

Alarm volume &Typical reduction rules

Degree of alarm

reduction

25%

50%

75%

90%

Automation

Page 41: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2942

FMX Know-how

FMX Starter Package– FMX introduction ws– Using the FMX tools training– FMX Rules templates– FMX Consultancy ws

Page 42: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2943

FMX Know-howFMX Solution Way of Working

Alarm Log Analysis – Shows what alarms to focus on first

Focus first on alarm reduction– often 50-70% alarm filtering possible

with little effort

Correlate alarms– Highlights the important problem hide

secondary problems

Automate– Implement current operations

procedures

CCITT7 SIGNALLING LINK FAILURE

23%

DIGITAL PATH QUALITY SUPERVISION

21%

DIGITAL PATH FAULT SUPERVISION

11%

RADIO X-CEIVER ADMINISTRATION MANAGED

OBJECT FAULT10%

DATA OUTPUT, AP DIRECT BLOCK OUTPUT, TRANSFER

FAULT6%

DIGITAL PATH UNAVAILABLE STATE FAULT

5%

RADIO X-CEIVER ADMINISTRATION BTS

EXTERNAL FAULT5%

SYNCHRONOUS DIGITAL PATH QUALITY SUPERVISION

3%

LAYER 2 FAULT2%

CCITT7 DISTURBANCE SUPERVISION LIMIT

REACHED2%

Page 43: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2944

Complete FMX solutions

Pre-defined solutions– Rule packages for Core (CNX)  and WCDMA (WRX) – FMX Surveillance expert integration &

configuration Customized solution

– FMX Rules on demand Efficient operations

– Requirement ws– A combination of the pre-defined and

customized solutions– Automation will be in focus for the FMX

rules design– FMX Solution support & LCM

Page 44: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2945

What to remember?

FMX do – lower OPEX,

– handle network growth without reducing quality,

– improve network availability

“If you can write it down, you can implement it

with FMX”

FMX do – lower OPEX,

– handle network growth without reducing quality,

– improve network availability

“If you can write it down, you can implement it

with FMX”

Page 45: Fault Management Automation, FMX (2G - 3G)

Top right corner for field-mark, customer or partner logotypes. See Best practice for example.

Slide title 40 pt

Slide subtitle 24 pt

Text 24 pt

Bullets level 2-520 pt

OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2946