HP & WIND Hellas IT
-
Upload
hp-software-solutions -
Category
Documents
-
view
836 -
download
1
description
Transcript of HP & WIND Hellas IT
©2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
Session ID: BTOT-WE-1630/4Speaker: Charis Tsevis, Senior Operations Mgr WIND Hellas SATwitter hashtag #HPSWU
2
WIND Hellas Company
History
• Established in 1992 as Telestet, the first mobile Telecom operator in Greece
• In 1993, places the first mobile call in Greece
• In Jan 2006 acquired Q-Telecom, the 4th mobile Telecom operator in the country
• In summer 2007 changes the brand name to WIND
• Oct 2007, acquired Tellas, fixed telephone and broadband Internet provider
• Tellas fully integrated in early 2010
Today
• Only Telecom Operator in Greece that offers all in one : Mobile, Fixed, Internet
• More than 5,5 million customers
• 99,6% network coverage across the country
• 400 WIND stores nationwide
• 1,07 billion Euro turn over in 2009
3
WIND IT Organization - I
• Business Support Services - BSS
Develop, Support and Implement all core business applications, such as :
- Postpaid Billing
- Prepaid Services
- Service Provisioning
- Point Of Sales
- CRM
- ERP
- Data Warehouse
- Fraud
• Business Solutions by strong vendors, such as :
- Ericsson, LHS
- Oracle (Siebel, BEA-WebLogic)
- SAP
- HP
- Microsoft
4
WIND IT Organization - II
• Operations Support Services - OSS
- Proactive systems support & maintenance
- 24x7 Monitoring & Reactive Support
- WIND Retail Chain support (400 shops across Greece)
- Provisioning of core infrastructure services
- Infrastructure software implementation & configuration (Openview, Legato,
EMC sw, etc.)
- 3 Data Centers, 300 Enterprise Systems, 250TB Storage Area Network
- Real time systems, mission critical applications
- Use of high-availability options, advanced technologies, Virtualization
• Solutions by strong vendors such as :
- HP
- Oracle, Sun
- EMC
- CISCO
- Microsoft
5
WIND IT Approach
6
Initiative – The 2004 story
• WIND decided in 2002 to introduce a new Prepaid platform
• Provide innovative services to our customer base
• Over 4.5M investment
• Revenue booster
• Real time system
• Platform became in production at Feb 2004
• 11 Clustered systems in Alpha Server, Tru64 environment
• Served 1.5M customers
• 12M monthly revenues
7
Daily Operational Challenges
We have to….
• Act as a service provider towards internal and external customers
• Meet tight SLAs (99,999% for Prepaid platform)
• Operate as a full 24x7 organization
• Manage Real time systems
So, we must…
• Maintain a robust and reliable event notification mechanism
• Utilize performance statistics and trend analysis results
• Focus on proactive activities and notification
• Enhance monitoring aspect to a wider range of activities and events, up to the
application level
• Keep a high-level of expertise
• Have efficient support contracts with vendors
Special challenges of the 2004 initiative
• Full manual monitoring cycle of the entire platform took 3hrs
• 6 months later the Athens Olympics 2004 were about to start
8
2004 Initiative – Implementation
• HP Operations Manager was chosen as the core monitoring platform
• HP Network Node Manager was chosen as network monitoring tool
• SPIs for Oracle, WebLogic, Tuxedo
• 250 processes to monitor
• 150 logfiles to check / filter
• MIBs for monitoring GSM network connectivity (SS7)
• Event notification through GUI, e-mail and SMS
• Define and classify events with severity types
• Define message groups and recipients
• Service Tree definition
9
Nowadays
• Significant growth of WND environment
Year Systems %Growth VirtualStorage
Capacity(TB) %Growth Monitors
2004 104 0 21 400
2005 128 23,1 0 37 76,2
2006 175 36,7 0 48 29,7
2007 188 7,4 0 75 56,2
2008 202 7,4 0 130 73,3
2009 253 25,2 30 210 61,5
2010 295 16,6 86 250 19,0 3500
• Systems increase from 2004 to 2010 : 283%
• Data growth from 2004 to 2010 : 1.190%
• Evolution of advanced technologies (Oracle RAC, Virtualization, J2EE)
• Rapid launch of new commercial products (3G / VAS services, SmartPhones evolution)
• No tolerance in service unavailability
10
Adapt IT Operations to new commercial needs
• Enhance and enrich the event notification environment
• Define better communication and escalation paths for incident management
• Measure the results and redefine the environment
• Next Steps : Monitor & Measure the core Business
11
Enhance & Enrich the event notification environment
• Not plain monitoring : “System up”, “System down”
• Introduce application monitoring based on
- Business Rules (e.g. specific action on MSISDN failed)
- Flow control (e.g. check for wrong delivery of Call Detail Records, flow inactivity over
time etc.)
- Control smooth application execution (message queues, concurrency of application
processes)
• Strict event classification and escalation path per event
• Integrate foreign monitoring systems (Ericsson SLM, Building Management System)
• Immediate checks, Smart checks (“no-login” checks for standby engineer assistance)
• Agent health check
• Service Tree Enhancement
12
Enhance & Enrich the event notification environment – Real Examples
• An alarm is raised with message “Action for MSISDN xxx Failed”
- If message appears once, then the customer performed an illegal action,
inform Customer Service dept to come in contact with customer.
- If message appears more than 100 times in a minute, raise CRITICAL,
check Provisioning platform
• Flow control check
- Check under directory </dir>
- Filenames with pattern “WSDP_*”
- If receive less than 2 files per 3 minutes, raise WARNING and on 3
WARNINGS raise CRITICAL
OR
- If file size is less than 2KB, raise CRITICAL alarm
• Message queue check
- Check provisioning queue length
- If length > threshold1, raise WARNING
- If length > threshold2, raise MAJOR
- If length > threshold3, raise CRITICAL
13
Immediate Checks / Smart Checks
• When an incident occurs, operator calls standby engineer
• Before logging in, standby engineer asks to perform some predefined checks / scripts
14
Build Service Tree
• Assist standby engineer to understand the service impact
15
Define better communication & escalation paths - I
System A System B System C
Network Device A
Network Device B
Network Device C
Corporate LAN/WAN
Database
Application 2
Application 1
Database
Application 2
Application 1
Database
Application 2
Application 1
System & Database Admins
Network Engineers
Application Engineers / A
Application Engineers / B
Application Engineers / C
Operator / Sys & DB cons
8 x 5 & Standby Engineers
Operator / Network cons
Operator / Application cons
24 x 7 Shift Operators
Openview ServerOM & NNM
Events / MessagesCritical, Major,Warning, Minor
Phone Call
Mail / SMS for Critical & Major Messages
All messages in group console GUI
16
Define better communication & escalation paths - Example
System A System B System C
Network Device A
Network Device B
Network Device C
Corporate LAN/WAN
Database
Application 2
Application 1
Database
Application 2
Application 1
Database
Application 2
Application 1
System & Database Admins
Network Engineers
Application Engineers / A
Application Engineers / B
Application Engineers / C
Operator / Sys & DB cons
8 x 5 & Standby Engineers
Operator / Network cons
Operator / Application cons
24 x 7 Shift Operators
Openview ServerOM & NNM
Events / MessagesCritical, Major,Warning, Minor
Phone Call Standby(non-working hours)
Phone Call Responsible(working hours)
17
Measure the results and redefine the environment
• Average No of events per day (Critical, Major, Warning) : 11.700
• Average response time of engineer to take over, Working hours : 5min
• Average response time of engineer to take over, Non-Working hours : 15min
Percentage of daily events per severity
18
Next Steps : Monitor & Measure the core Business
• Beta testing of OMi9
- Selected by the HP-Boeblingen team as candidates for beta testing
- Provide feedback to HP for improvements
• Pilot run, proof of concept for EUM
- Improve quality of service towards our retail chain
- Infrastructure monitoring already in place through OM
- Focus on end-user-experience and measure core business transactions
19
OMi9 - Feedback
Main features, areas of interest
• Unified environment, common look and feel for the entire BSM product suite (APM, EUM,
OMi, etc)
• Intuitive and well-designed GUI
• UCMDB, CIs
• Dynamic update of CIs
• Association of events with CIs
• Correlation Rules -> Focus on root cause of a problem
• Flexible and efficient graphical representation of landscape on top of the
Event Management Foundation (the well-known Operations Manager) :
- Site Map
- Topology View, TBEC
- Health Perspective View, same with BAC
- Performance Perspective
• Integration with BAC, SiteScope, Service Desk
OMi9 – GUI environment
21
EUM – Proof Of Concept
Scope
• Measure core-business transactions exactly as the end-users experience them
• Increase proactive actions
• Reduce service unavailability
• Proceed in corrective actions, based on findings, to improve quality of service
Implementation
• Selected 3 different shops across Greece
• Defined core transactions to monitor
• Environment setup and configuration
• Record business transactions (BPM)
• Deploy monitors
• Create baseline
• Start real monitoring & measurement
22
BAC Results - I
23
BAC Results - II
24
BAC Results - III
25
Benefits
• Increased availability times
• More proactive than reactive
• Kept low headcount in Operations
• Reduced emergency overtimes of standby Operations personnel
• Better systems resource utilization according to trend analysis through OVPM
• Decreased Fraud cases due to increased availability times (revenue gain)
• Decreased negative prepaid balances due to increased availability of Prepaid systems
• Improved company image due to increased availability times
• EUM results convinced company to invest on this platform
26
Roadmap
• Enhance application monitoring to include more business logic
• Upgrade to OMi9/NNMi
• Invest on a bundle of HP software products such as
- Service Manager, Service Discovery, Asset Manager, Program Manager, BPM/RUM
- Service Level Management, Change Management, Server Automation
- VMware Infrastructure SPI
• Under negotiations with HP to produce an Enterprise License Agreement for the above
products
Continue the conversation with your peers at the HP Software Community hp.com/go/swcommunity