Fault Management Automation, FMX (2G - 3G)
-
Upload
sarah-love -
Category
Documents
-
view
30 -
download
0
description
Transcript of Fault Management Automation, FMX (2G - 3G)
Slide titleIn CAPITALS
50 pt
Slide subtitle 32 pt
Fault Management AutomationFMX
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-292
Presentation agenda
Overview Challenges in Supervision with FMX Experience from other markets FMX Services and products Alarm log analysis GSM/CN Network Alarm log analysis WCDMA Network Rule Development process FMX Know-how Complete FMX solutions
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-293
Overview
OSS-RC
FM(FMX) PM CM SM
Telecom Network
NE
NE
NE
NE
NE
NE NE
NE
NENE
2G3G
CORE
IMS
TeMIP/Netcool (NMX)
NMS system
Trouble Ticketing
Slide titleIn CAPITALS
50 pt
Slide subtitle 32 pt
Do you have a clear view?
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-295
Challenges in supervision
Poor visibility in supervision – Alarm storms problem– Frequent alarms– Several alarms indicate the same problem
CCITT7 SIGNALLING LINK FAILURE
DIGITAL PATH FAULT SUPERVISION
DIGITAL PATH QUALITY SUPERVISION
CELL LOGICAL CHANNEL AVAILABILITYSUPERVISIONRADIO X-CEIVER ADMINISTRATIONMANAGED OBJECT FAULT
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-296
Challenges in supervision
Poor visibility in supervision – Alarm storms problem– Frequent alarms– Several alarms indicate the same problem
Network growth with same or reduced staff OPEX reduction Improvement of network availability
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-297
Way forward
Increase visibility
Increase visibility
Show problem
Show problem
Automatic diagnostics &
repair
Automatic diagnostics &
repair
OPEX reductionHandle network growth
Network availability improved
OPEX reductionHandle network growth
Network availability improved
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-298
Automatic diagnostics &
repair
Automatic diagnostics &
repair
Get a clear view with FMX
Increase visibility
Increase visibility
Show problem
Show problem
Prioritize the alarmsFilter short lived alarms
Alarm enrichment
Prioritize the alarmsFilter short lived alarms
Alarm enrichment
Execute repair sequence Alarm enrichment (db´s,mml)
Create Trouble Tickets
Execute repair sequence Alarm enrichment (db´s,mml)
Create Trouble Tickets
Hide secondary alarmsShow problem, not alarms (IUB down)
Create frequent alarms
Hide secondary alarmsShow problem, not alarms (IUB down)
Create frequent alarms
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-299
Scenario 1 – too many alarms
3G Network with 40.000 alarms/day
Small operational staff
Hard to distinguish between important problems and noise
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2910
Scenario 1 - FMX Solution
FMX Filter and correlation rules
Process 86% of all alarms
Filter out 67% of total alarm flow
Remaining 13.200 Alarms/Day of 40.000->Time saved 1-3 hours / day & operator
Increase visibility
Increase visibility
Show problem
Show problem
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2911
Scenario 2 – automatic information gathering
2G Network with 20.000 alarms/day
2800 Radio alarms from BTS(RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT)
40% Critical/A1
Operators has to send MML command to AXE to retrieve fault codes
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2912
Scenario 2 – FMX Solution
FMX send command to the switches for all Critical alarms
Fault codes are translated into text and presented in the alarm text
FMX sends 1120 commands/day
->Time saved >9 hours / day
Increase visibility
Increase visibility
Automatic diagnostics &
repair
Automatic diagnostics &
repair
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2913
Scenario 3 – Automatic actions
2G Network with 8.500 alarms/day
8% Radio alarms from BTS(RADIO X_CEIVER ADMINISTRATION MANAGED OBJECT FAULT)
24% Time slot fault (TS SYNC FAULT)
Usually block and de-block of the time slot solves the problem
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2914
Scenario 3 – FMX Solution
FMX send commands to the switches for all time slot alarms(RXBLI and RXBLE)
FMX block and de-block 338 timeslots/day
->Time saved 3-4 hours/day
Automatic diagnostics &
repair
Automatic diagnostics &
repair
Increased scope:
Info number percentCELL LOGICAL 2507Filtered 1120 45%Solved 1167 47%Alarm updated 220 8%
-> Increased scope >50 hours/day
Increased scope:
Info number percentCELL LOGICAL 2507Filtered 1120 45%Solved 1167 47%Alarm updated 220 8%
-> Increased scope >50 hours/day
Experience from other markets
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2915
Show me the money
Assumption: – 2G network for large operator– Automation of the network using FMX
Result:– Unavailability reduced ( 70%) – 5.400 man hours saved / year– Solution management contracts
Experience from other markets
Slide titleIn CAPITALS
50 pt
Slide subtitle 32 pt
FMX Services and products
Complete FMX solutions and FMX know-how
FMX Service description
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2917
FMX Service options flow
Alarm Log
Analysis
FMX Rules
on Demand
FMX Surveillance expert integration and configuration
FMX Starter package
(CNX, WRX)
Solution Support & LCM
SolutionManagement
SolutionIntegration
Solution Analysis
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2918
Alarm log analysisSummary for GSM/CN Network
November 2008, 16 and 21 days (madmas2o & bnamas2o)
2.605.945 alarms received:– 1.293.063 alarms on Madrid-OSS (madmas2o)– 1.312.882 alarms on Barcelona-OSS (bnamas2o)
71.667 alarms per day:– 80.816 alarms per day on Madrid-OSS (madmas2o)– 62.518 alarms per day on Barcelona-OSS (bnamas2o)
Results for GSM/CN Network
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2919
Alarm log analysisTop 10 frequent alarms
Results for GSM/CN Network
The 10 most frequent alarms mean the 67% of total alarms
SpecificProblemTotal number of
alarms
Percent of total alarm
flow
RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT 767287 29%
RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT 250324 10%
TEST SYSTEM ACTIVATED 164813 6%
CCITT7 SIGNALLING LINK FAILURE 139105 5%
DIGITAL PATH FAULT SUPERVISION 109838 4%
DIGITAL PATH QUALITY SUPERVISION 96000 4%
EXTERNAL ALARM 83782 3%
FSC-SS7Manager/B/2 : MTPL-3: Link Out Of Service 83113 3%
DIGITAL PATH UNAVAILABLE STATE FAULT 27716 1%
HEARTBEAT FAILURE 26585 1%
TOTAL 1748563 67%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2920
Alarm log analysisTop 10 short-live* alarms
Results for GSM/CN Network
The 51% of alarms are cleared in less than 5 minutes
*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )
SpecificProblemTotal
number of alarms
Total number of shortlived
alarms
Shortlived percentage
Shortlived percentage of all alarms
RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT 767287 596415 78% 23%
TEST SYSTEM ACTIVATED 164813 162903 99% 6%
CCITT7 SIGNALLING LINK FAILURE 139105 131225 94% 5%
RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT 250324 111774 45% 4%
DIGITAL PATH FAULT SUPERVISION 109838 98515 90% 4%
FSC-SS7Manager/B/2 : MTPL-3: Link Out Of Service 83113 83002 100% 3%
EXTERNAL ALARM 83782 72417 86% 3%
DIGITAL PATH QUALITY SUPERVISION 96000 24616 26% 1%
HEARTBEAT FAILURE 26585 23761 89% 1%
DIGITAL PATH UNAVAILABLE STATE FAULT 27716 20935 76% 1%
TOTAL 1748563 1325563 51%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2921
Alarm log analysisSuggested FMX Rules (I)
Results for GSM/CN Network
Blacklisting of nodes Filter out all the alarms from the nodes under maintenance in the
blacklist
RADIO X-CEIVER ADMINISTRATION BTS EXTERNAL FAULT Single slogan presented (MAINS FAIL, DOOR OPEN and SITE
ON BATTERY), Filters short-lived, Filters unimportant alarms and Frequent alarms created.
Short-lived filter will reduce the alarm flow with on average ~23%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2922
Alarm log analysisSuggested FMX Rules (II)
Results for GSM/CN Network
RADIO X-CEIVER ADMINISTRATION MANAGED OBJECT FAULT Filters short-lived and Create FREQUENT alarm Automatic block/deblock on TS Presents DIP & STATE if OML FAULT Gets affected CELL to enable correlation with the service alarm CELL
LOGICAL CHANNELS AVAILABILITY SUPERVISION RXMFP is sent, the fault codes are translated to text and presented
with the alarm. An alarm reduction of ~4% filtering short-lived alarms only
DIGITAL PATH FAULT SUPERVISION Filter short-lived, Frequent alarms created Correlation with the DIGITAL PATH UNAVAILABLE STATE FAULT
alarm so that if both alarms appear on the same MO and DIP only one of them are shown.
An alarm reduction of ~4% filtering short-lived alarms only
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2923
Alarm log analysisSuggested FMX Rules (III)
Results for GSM/CN Network
DIGITAL PATH QUALITY SUPERVISION Will be automatically reset by the rule (DTQSR), i.e.maximum 3 times
on the same DIP within 24 hours If alarm comes 3 days in a row, same DIP, a FREQUENT error
message is created
CCITT7 SIGNALLING LINK FAILURE Filter short-lived, Frequent alarms created Not short-lived can be analyzed to find the related DIP, DIP-STATE and
if applicable the route-name (R). An alarm reduction of ~5% filtering short-lived alarms only
EXTERNAL ALARM Filters short-lived with non important alarms slogans Frequent alarms created for non important slogans
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2924
Alarm log analysisSuggested FMX Rules (IV)
Results for GSM/CN Network
TEST SYSTEM ACTIVATED Unless the operators keep track of whether a test system is active, a
reduction of the total alarm flow of about ~6%
FSC-SS7Manager/B/2: MTPL-3: Link Out Of Service Filter short-lived, Since most of these alarms are short-lived (~99%) An alarm reduction of ~3% filtering short-lived alarms only
Filtering short-lived alarms only 51% of total alarm flow can be cleared
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2925
Alarm log analysisSummary for WCDMA Network
December 2008, 18 days (Coruña rcmser01 & Leganés rcmser02)
1.207.633 alarms received:– 774.387 alarms on Coruña-OSS (rcmser01)– 433.246 alarms on Leganés-OSS (rcmser02)
67.090 alarms per day:– 43.021 alarms per day on Coruña-OSS (rcmser01)– 24.069 alarms per day on Leganés-OSS (rcmser02)
Results for WCDMA Network
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2926
Alarm log analysisTop 20 frequent alarms Coruña-OSS (rcmser01)
Results for WCDMA Network
Specific Problem Object Type Number %
Ip over ATM Link Supervision Recovered URAN 320988 41%
Ip over ATM Link Supervision Failed URAN 320734 41%
AntennaBranch_AntennaSystemProblemInBranchA URAN 12786 2%
AlmDevice_ExternalAlarm URAN 10567 1%
DbccDevice_GammaDownlinkFailure URAN 9625 1%
IMA Link Transmit Unusable at Far End URAN 8326 1%
IMA Group Insufficient Links at Far End URAN 7725 1%
Heartbeat Failure URAN 7666 1%
NodeSynchTp_ConnectionLost URAN 7570 1%
UtranCell_NbapMessageFailure URAN 7268 1%
UtranCell_ServiceUnavailable URAN 5874 1%
IMA Link Reception Unusable at Far End URAN 5843 1%
IMA Group Insufficient Links URAN 4906 1%
Remote Defect Indication on IMA Link URAN 4372 1%
NssSynchronization_SynchRefChanged URAN 4239 1%
NssSynchronization_SystemClockStatusChanged URAN 4186 1%
PDH Alarm Indication Signal URAN 3364 0%
IMA Group Configuration Aborted URAN 2873 0%
RruDeviceGroup_FanFailure URAN 1946 0%
NbapDedicated_RncRbsControlLinkDown URAN 1612 0%
Total 752470 97%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2927
Alarm log analysisTop 20 frequent alarms Leganés-OSS (rcmser02)
Results for WCDMA Network
Specific Problem Object Type Number %
Ip over ATM Link Supervision Recovered URAN 63284 15%
Ip over ATM Link Supervision Failed URAN 62801 14%
AlmDevice_ExternalAlarm URAN 60214 14%
UtranCell_ComMeasFailCongCon URAN 31318 7%
IMA Link Reception Unusable at Far End URAN 24278 6%
IMA Link Transmit Unusable at Far End URAN 21346 5%
Heartbeat Failure URAN 16769 4%
Remote Defect Indication on IMA Link URAN 14146 3%
AntennaBranch_AntennaSystemProblemInBranchA URAN 11450 3%
NssSynchronization_SynchRefChanged URAN 9292 2%
IMA Group Insufficient Links URAN 8921 2%
NssSynchronization_SystemClockStatusChanged URAN 8309 2%
UtranCell_ServiceUnavailable URAN 7441 2%
NodeSynchTp_ConnectionLost URAN 7187 2%
UtranCell_NbapMessageFailure URAN 7141 2%
PDH Alarm Indication Signal URAN 6041 1%
DbccDevice_GammaDownlinkFailure URAN 5371 1%
IMA Group Insufficient Links at Far End URAN 5231 1%
OpticalInterfaceLink_OpticalInterfaceLinkFailure URAN 4803 1%
NodeSynch_Phase_Difference_Measurement_Failed URAN 4549 1%
Total 379892 88%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2928
Alarm log analysisTop 10 short-live* alarms Coruña-OSS (rcmser01)
Results for WCDMA Network
*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )
SpecificProblem Object TypeTotal
number
Total number shortlived with duration less
than 5 minutes.
Percentage of
shortlived
Percent of Shortlived
of total
DbccDevice_GammaDownlinkFailure URAN 9625 8834 92% 1%
AntennaBranch_AntennaSystemProblemInBranchA URAN 12786 8801 69% 1%
AlmDevice_ExternalAlarm URAN 10567 8117 77% 1%
Ip over ATM Link Supervision Recovered URAN 320988 8095 3% 1%
Ip over ATM Link Supervision Failed URAN 320734 7851 2% 1%
IMA Link Transmit Unusable at Far End URAN 8326 7828 94% 1%
IMA Group Insufficient Links at Far End URAN 7725 7492 97% 1%
NodeSynchTp_ConnectionLost URAN 7570 6935 92% 1%
UtranCell_NbapMessageFailure URAN 7268 6878 95% 1%
Heartbeat Failure URAN 7666 5440 71% 1%
Total 713255 76271 10% 10%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2929
Alarm log analysisTop 10 short-live* alarms Leganés-OSS (rcmser02)
Results for WCDMA Network
*The term “short-lived alarm” means that the alarm is cleared within a short time (5 minutes) after the alarm was issued (Candidates for alarm filtering )
SpecificProblemObject Type
Total number
Total number shortlived
with duration less than 5 minutes.
Percentage of
shortlived
Percent of Shortlived
of total
AlmDevice_ExternalAlarm URAN 60214 34617 57% 8%
UtranCell_ComMeasFailCongCon URAN 31318 30974 99% 7%
Ip over ATM Link Supervision Recovered URAN 63284 24004 38% 6%
Ip over ATM Link Supervision Failed URAN 62801 23920 38% 6%
IMA Link Reception Unusable at Far End URAN 24278 13499 56% 3%
IMA Link Transmit Unusable at Far End URAN 21346 12108 57% 3%
Remote Defect Indication on IMA Link URAN 14146 9212 65% 2%
Heartbeat Failure URAN 16769 8259 49% 2%
UtranCell_NbapMessageFailure URAN 7141 6307 88% 1%
AntennaBranch_AntennaSystemProblemInBranchA URAN 11450 6108 53% 1%
Total 312747 169008 54% 39%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2930
Alarm log analysisSuggested FMX Rules (I)
Results for WCDMA Network
Module Handled Alarms (Italic = FMX created) WRX_IUB_disturbances UtranCell_ServiceUnavailable IUB down FREQUENT IUB disturbances Fach_InternalResourceUnavailable Pch_InternalResourceUnavailable Rach_InternalResourceUnavailable NbapDedicated_RncRbsControlLinkDown FREQUENT NbapDedicated_RncRbsControlLinkDown WRX_NodeB_disturbances UtranCell_NbapMessageFailure UtranCell_RbsFailure NodeB down FREQUENT NodeB disturbances WRX_AuxPlugInUnit_PiuConnectionLost AuxPlugInUnit_PiuConnectionLost FREQUENT AuxPlugInUnit_PiuConnectionLost WRX_NodeSynchTp_ConnectionLost NodeSynchTp_ConnectionLost FREQUENT NodeSynchTp_ConnectionLost WRX_PlugInUnitFault Plug-In Unit Fault FREQUENT Plug-In Unit Fault PlugInUnit_BoardSwitchCoreFault FREQUENT PlugInUnit_BoardSwitchCoreFault SwitchModule_SwitchPlaneAFault
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2931
Alarm log analysisSuggested FMX Rules (II)
Results for WCDMA Network
Module Handled Alarms (Italic = FMX created) WRX_UtranCell_ComMeasFailCongCon UtranCell_ComMeasFailCongCon FREQUENT UtranCell_ComMeasFailCongCon WRX_Utran_External_Alarm AlmDevice_ExternalAlarm FREQUENT AlmDevice_ExternalAlarm WRX_VCeteAlarmIndicationSignal VC ete Alarm Indication Signal FREQUENT VC ete Alarm Indication Signal WRX_CchFrameSynch_TimingAdjCtrlFrame CchFrameSynch_TimingAdjCtrlFrame FREQUENT CchFrameSynch_TimingAdjCtrlFrame WRX_Clu_LossOfMains Clu_LossOfMains FREQUENT Clu_LossOfMains Blocking Filters NbapCommon_RncRbsControlLinkDown NbapDedicated_RncRbsControlLinkLossOfRedundancy NbapCommon_RncRbsControlLinkLossOfRedundancy NbapCommon_SuccessfulReconfigurationAtFailure NbapCommon_SuccessfulReconfigurationOrdered NbapDedicated_SuccessfulReconfigurationAtFailure NbapDedicated_SuccessfulReconfigurationOrdered SYSTEM_CLOCK_IN_HOLDOVER_MODE System Clock in Holdover Mode E1Ttp_NOCRC4MFA NssSynchronization_SystemClockStatusChanged NodeBFunction_O&MapplicationRestarted SlotMo New PIU detected, matches the defined configuration AscDeviceGroup_SuccessfulRecoveryActionPerformedBoardRestart
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2932
Alarm log analysisEstimated Alarm Reduction Coruña-OSS (I)
Results for WCDMA Network
Specific Problem Number % Type of filterEst. % alarm
reduction Filtered
Ip over ATM Link Supervision Recovered 320988 41%Filter with WRX Block 100% 320988
Ip over ATM Link Supervision Failed 320734 41%Filter with WRX Block 100% 320734
AntennaBranch_AntennaSystemProblemInBranchA 12786 2% 0
AlmDevice_ExternalAlarm 10567 1%correlation, short-lived filter, frequent 77% 8137
DbccDevice_GammaDownlinkFailure 9625 1% 0
IMA Link Transmit Unusable at Far End 8326 1% 0
IMA Group Insufficient Links at Far End 7725 1% 0
Heartbeat Failure 7666 1% 0
NodeSynchTp_ConnectionLost 7570 1% short-lived filter, frequent 92% 6964
UtranCell_NbapMessageFailure 7268 1%correlation, short-lived filter, frequent 95% 6905
UtranCell_ServiceUnavailable 5874 1%correlation, short-lived filter, frequent 49% 2878
IMA Link Reception Unusable at Far End 5843 1% 0
IMA Group Insufficient Links 4906 1% 0
Remote Defect Indication on IMA Link 4372 1% 0
NssSynchronization_SynchRefChanged 4239 1%Filter with WRX Block 100% 4239
Alarm is candidate for Rules on Demand package
Alarm is processed by WRX package
No defined FMX package for alarm
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2933
Alarm log analysisEstimated Alarm Reduction Coruña-OSS (II)
Results for WCDMA Network
Specific Problem Number % Type of filterEst. % alarm
reduction Filtered
NssSynchronization_SystemClockStatusChanged 4186 1% Auto-ack/ FMX filter 100% 4186
PDH Alarm Indication Signal 3364 0% 0
IMA Group Configuration Aborted 2873 0% 0
RruDeviceGroup_FanFailure 1946 0% 0
NbapDedicated_RncRbsControlLinkDown 1612 0% Auto-ack/ FMX filter 100% 1612
AuxPlugInUnit_SuccessfulRecoveryActionPerformed 1575 0% 0
OpticalInterfaceLink_OpticalInterfaceLinkFailure 1219 0% 0
PacketDataRouter_CnNotRespondingToGTPEcho 1153 0% 0
E1Ttp_NOCRC4MFA 1001 0% Auto-ack/FMX-filter 100% 1001
PDH Remote Defect Indication 938 0% 0
Rach_InternalResourceUnavailable 917 0%correlation, short-lived
filter, frequent 97% 889
Fach_InternalResourceUnavailable 836 0%correlation, short-lived
filter, frequent 98% 819
Pch_InternalResourceUnavailable 825 0%correlation, short-lived
filter, frequent 98% 809
Loss of Cell Delineation 822 0% 0
PDH Loss of Signal 812 0% 0
Total 762568 98% 87% 680161
Alarm is candidate for Rules on Demand package
Alarm is processed by WRX package
No defined FMX package for alarm
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2934
Alarm log analysisEstimated Alarm Reduction Leganés-OSS (I)
Results for WCDMA NetworkAlarm is candidate for Rules on Demand package
Alarm is processed by WRX package
No defined FMX package for alarm
Specific Problem Number % Type of filterEst. % alarm
reduction Filtered
Ip over ATM Link Supervision Recovered 63284 15%Filter with WRX Block 100% 63284
Ip over ATM Link Supervision Failed 62801 14%Filter with WRX Block 100% 62801
AlmDevice_ExternalAlarm 60214 14%correlation, short-lived filter, frequent 57% 34322
UtranCell_ComMeasFailCongCon 31318 7%correlation, short-lived filter, frequent 99% 31005
IMA Link Reception Unusable at Far End 24278 6% 0
IMA Link Transmit Unusable at Far End 21346 5% 0
Heartbeat Failure 16769 4% 0
Remote Defect Indication on IMA Link 14146 3% 0
AntennaBranch_AntennaSystemProblemInBranchA 11450 3% 0
NssSynchronization_SynchRefChanged 9292 2%Filter with WRX Block 100% 9292
IMA Group Insufficient Links 8921 2% 0
NssSynchronization_SystemClockStatusChanged 8309 2%Auto-ack/ FMX filter 100% 8309
UtranCell_ServiceUnavailable 7441 2%correlation, short-lived filter, frequent 38% 2828
NodeSynchTp_ConnectionLost 7187 2% short-lived filter, frequent 58% 4168
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2935
Alarm log analysisEstimated Alarm Reduction Leganés-OSS (II)
Results for WCDMA NetworkAlarm is candidate for Rules on Demand package
Alarm is processed by WRX package
No defined FMX package for alarm
Specific Problem Number % Type of filterEst. % alarm
reduction Filtered
UtranCell_NbapMessageFailure 7141 2%correlation, short-lived filter, frequent 88% 6284
PDH Alarm Indication Signal 6041 1% 0
DbccDevice_GammaDownlinkFailure 5371 1% 0
IMA Group Insufficient Links at Far End 5231 1% 0
OpticalInterfaceLink_OpticalInterfaceLinkFailure 4803 1% 0
NodeSynch_Phase_Difference_Measurement_Failed 4549 1% 0
NodeBFunction_NodeRestarted 4526 1% 0
PDH Remote Defect Indication 4318 1% 0
IMA Group Configuration Aborted 4069 1% 0
UtranCell_ComMeasFailAdmCon 2793 1% 0
Pch_InternalResourceUnavailable 2712 1%correlation, short-lived filter, frequent 87% 2359
Fach_InternalResourceUnavailable 2646 1%correlation, short-lived filter, frequent 86% 2276
Rach_InternalResourceUnavailable 2611 1%correlation, short-lived filter, frequent 85% 2219
NodeBFunction_MpTrafficApplicationRestarted 2410 1% 0
IMA Link Transmission Misconnected 2322 1% 0
ObifDeviceGroup_PreloadFailed 2305 1% 0
Total 410604 95% 53% 229147
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2936
Alarm log analysisEstimated Alarm Reduction
Results for WCDMA Network
With the WRX rules package, an estimated 87% and 53% of total alarm flow reduction may be achieved on
Coruña-OSS and Leganés-OSS respectively
The reduction level in the table is based on 5 minutes filtering for short lived alarms
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2937
Alarm log analysisRules on Demand
Results for WCDMA Network
By using simple short-lived filtering, a further 4% and 11% of total alarm flow reduction may be achieved on Coruña-OSS and Leganés-OSS .
The focus would be on the IMA alarms group:– IMA Link Transmit Unusable at Far End– IMA Group Insufficient Links at Far End– IMA Link Reception Unusable at Far End– IMA Group Insufficient Links– Remote Defect Indication on IMA Link– IMA Group Configuration Aborted
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2938
Rule Development ProcessWhat can we do with FMX?
“If you can write it down, you can
implement it with FMX”
“If you can write it down, you can
implement it with FMX”
!
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2940
Rule Development ProcessAvailable features used
Rules are grouped in independent modules All controllable parameters are set in a text file, can be
updated in run-time Rule statistics available
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2941
Blocking Filters Threshold filters
Alarm-based correlation
Duration filters
Topology-based
correlation
Effort to implement reduction
Rule Development ProcessEffect vs Effort
Alarm volume &Typical reduction rules
Degree of alarm
reduction
25%
50%
75%
90%
Automation
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2942
FMX Know-how
FMX Starter Package– FMX introduction ws– Using the FMX tools training– FMX Rules templates– FMX Consultancy ws
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2943
FMX Know-howFMX Solution Way of Working
Alarm Log Analysis – Shows what alarms to focus on first
Focus first on alarm reduction– often 50-70% alarm filtering possible
with little effort
Correlate alarms– Highlights the important problem hide
secondary problems
Automate– Implement current operations
procedures
CCITT7 SIGNALLING LINK FAILURE
23%
DIGITAL PATH QUALITY SUPERVISION
21%
DIGITAL PATH FAULT SUPERVISION
11%
RADIO X-CEIVER ADMINISTRATION MANAGED
OBJECT FAULT10%
DATA OUTPUT, AP DIRECT BLOCK OUTPUT, TRANSFER
FAULT6%
DIGITAL PATH UNAVAILABLE STATE FAULT
5%
RADIO X-CEIVER ADMINISTRATION BTS
EXTERNAL FAULT5%
SYNCHRONOUS DIGITAL PATH QUALITY SUPERVISION
3%
LAYER 2 FAULT2%
CCITT7 DISTURBANCE SUPERVISION LIMIT
REACHED2%
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2944
Complete FMX solutions
Pre-defined solutions– Rule packages for Core (CNX) and WCDMA (WRX) – FMX Surveillance expert integration &
configuration Customized solution
– FMX Rules on demand Efficient operations
– Requirement ws– A combination of the pre-defined and
customized solutions– Automation will be in focus for the FMX
rules design– FMX Solution support & LCM
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2945
What to remember?
FMX do – lower OPEX,
– handle network growth without reducing quality,
– improve network availability
“If you can write it down, you can implement it
with FMX”
FMX do – lower OPEX,
– handle network growth without reducing quality,
– improve network availability
“If you can write it down, you can implement it
with FMX”
Top right corner for field-mark, customer or partner logotypes. See Best practice for example.
Slide title 40 pt
Slide subtitle 24 pt
Text 24 pt
Bullets level 2-520 pt
OPER/MUIB-09:004418 Uen Rev A Ericsson Internal Fault Management Automation (2G & 3G) 2009-04-2946