Post on 19-Aug-2020
Mainframe Performance Management
Can Make the Dinosaur Dance
AUTHORS Srinivas Murthy Potharaju, ETS Mainframe CoE HCL Technologies, Hyderabad Dr Usha Thakur, ATS Technical Research HCL Technologies, Chennai
October, 2010
Mainframe Performance Management: Can Make the Dinosaur Dance
© 2010, HCL Technologies Ltd.
Mainframe Performance Management 2
Mainframe Performance Management 3
Contents Introduction .................................................................................................................................................. 4
Purpose ......................................................................................................................................................... 4
Significance of Mainframes ........................................................................................................................... 4
Drivers for Mainframe Performance............................................................................................................. 5
Mainframe Performance Services from HCL................................................................................................. 6
HCL Approach ............................................................................................................................................ 7
Benefits ....................................................................................................................................................... 10
Endnotes ..................................................................................................................................................... 12
Mainframe Performance Management 4
Introduction There was a time when IT folks could show value by focusing only on operational performance. Today, however, the focus has shifted more to showing the extent to which IT products and services create value and contribute to their organization’s business objectives. Businesses want to know how IT can help in expanding their company’s customer base, increasing products and services portfolios, increasing the profit margin, tapping into new markets, identifying activities for outsourcing, and much more. The alignment of IT activities and deliverables to business goals gains even more significance in the world of mainframes, given the high investments associated with them.
It is no longer enough to show mainframe performance metrics only in terms of operational efficiency; we need metrics that can reflect the extent to which mainframes help their organization fulfil business goals. That this is not just a fad is very clear from a study undertaken recently by Gartner and Financial Executives International (FEI). According to that study a fairly high number of CIOs are already reporting to CFOs, and the latter have a significant say in all IT investment matters.1
Purpose The audience of this paper are CIOs, CEOs, CFOs, and other executives whose business operations are heavily dependent upon the performance of mainframes. It examines the main drivers for mainframe performance management by organizations that have decided to continue using them. This paper also highlights key features of the performance management service from HCL.2
Significance of Mainframes Throughout the history of the mainframe, IBM has retained its dominance,3 though not without its share of controversy.4 It is estimated that IBM’s mainframe sales will reach $3.3 billion in 2010, which would represent about three‐quarters of the overall mainframe market of $4.5 billion.5
For sometime now, there has been a popular belief that mainframes are living on borrowed time and may not be around in the near future. Writing in 2003, John R. Phelps of Gartner pointed out that a number of developments over the past 15 years6 led to stories about the death of the IBM mainframe, but the reality has been to the contrary. Currently, a great deal of attention is being given to the declining revenue from mainframes and servers in general,7 and once again we are seeing stories about the end of mainframes in the very near future. What is interesting, however, is that IBM has continued to modernize and invest in a range of mainframes, improve pricing, and reach out to new customers, especially in newly industrializing economies,8 which indicates that IBM still has confidence in continued business from mainframes. In fact, IBM is solidifying its position in niche accounts. For instance, the
Mainframe Performance Management 5
zEnterprise system (z196)‐‐latest mainframe model from IBM‐‐is targeted at the top 10% to 20% of its customers where the volume of Millions of Instructions Per Second (MIPS) is very high.9 MIPS is one of the most important criterion that IBM uses for calculating the cost of mainframe license.10
The mainframe story would be incomplete without its customers. Although some customers are on the look out to reduce dependency on mainframes by moving to systems powered by newer technologies, many still continue to use them for historical reasons. Mainframe customers are big companies in data‐intensive industries such as distribution, financial services, general business, and public sector that invested in mainframes several years ago. For these customers, reliability, security, and scalability are top priorities.11 There is a general consensus that any improvement in the end‐to‐end performance of mainframes is a very important step in bringing down the cost of owning and running mainframes.
Drivers for Mainframe Performance In general, the main business drivers for improving the performance of mainframes are as follows:
Reduce the Total Cost of Ownership (TCO) of mainframe servers
Increase productivity
Make the online applications available 24/7 to an increasing number of customers without any delay12
Improve end‐user experience e.g., comparing auto insurance quotes13 and the like transactions
Increase customer retention14
Eliminate SLA violation
Assuming that IT is a means for achieving the business objectives then it is imperative for IT management and staff to be on the same page as the business management and staff in their company. This will not only give all IT activities and decisions a business context but also reveal their significance and impact on business.
The changing business environment today is compelling IT management to look at ways of obtaining maximum performance from the core mainframe applications and of aligning applications directly to the overall business objectives of their organization. The bulk of the mainframe applications were developed years ago, which have since then undergone enhancements or patchwork. Consequently, they have lost maintainability and traceability to business requirements.
Today, more than ever before, there is an ever changing workload being executed on the mainframe. Starting with traditional batch and online workloads, there is an increasing number of transactions
Mainframe Performance Management 6
running on the mainframe which are invoked from the Web. Additionally, there are queries that are used for complex reporting and analytics in Data Warehouse & Business Intelligence environments.
Poor mainframe performance may be caused by all or a combination of the following problems:
Environment, system and sub‐system parameters are not set correctly
Aging and monolithic applications are large and complex, and therefore difficult to integrate with modern systems, which are distributed and agile
Database parameters are not tuned for optimal performance
Applications and databases are poorly designed
Application code and SQLs are not tuned for performance
Online applications may not be available due to the scheduling of jobs in critical path and end‐of‐day batch jobs
File block size and buffer parameters are not set correctly, leading to prolonged and high usage of CPU and I/O
The above‐mentioned technical problems need to be addressed by IT teams, given that if they are left unattended they will most likely lead to the following undesirable consequences:
Higher licensing cost of mainframes resulting from a high consumption of MIPS by applications
Increasing customer dissatisfaction and decline in customer because of poor response time of online applications
Lower productivity of day time employees caused by extended batch windows cutting into regular business hours
SLA violations due to the inability of applications to scale up to increasing data volumes
Mainframe Performance Services from HCL This is a comprehensive service from HCL. Using proven methodologies and frameworks, it identifies poorly designed processes, programs, databases, and environment on the mainframe. On the basis of its findings, HCL proposes a solution and remedial strategy. During implementation, HCL enhances the configuration of the mainframe processes, programs, databases, and services such that the overall response and execution time is faster than before. Customers who have availed of this service have been able to reduce MIPS usage by their applications, lower TCO, and utilize the available processing capacity for newer workloads.
HCL Approach As illustrated in Figure 1, HCL uses its proprietary framework for workload optimization to help customers analyze their workload and identify areas of application tuning. This approach takes into account inputs on the architecture of business applications and databases, the design of online and batch components, and also usage/runtime statistics (from SMF & RMF, performance management tools, history reports, and environment as well system/ subsystem parameters).
Figure 1: HCL Proprietary Framework
As highlighted below, the objective of the assessment approach adopted by HCL is to identify poorly performing workloads on the mainframe so that appropriate recommendations could be made and a remedial strategy developed.
PERFORMANCE OPTIMIZATION ASSESSMENT APPROACH
As shown in Figure 2, HCL first collects data on various parameters from the client systems and subsystems. It analyzes that data and then proposes a solution, together with a remedial strategy.
Mainframe Performance Management 7
Propose Solutions
Remediate
Make recommendations to optimize performance of the workload. Estimate cost, effort and duration required for implementing proposed recommendations aimed at optimizing performance. Prepare Gain Vs Effort matrix to prioritize each recommendation.
Construct high level designs for each approved recommendation.
Conduct baseline performance of the code (production version) in controlled test region. Carry out functional and performance tests of impacted components in controlled test environment. Compare results from performance test, implement agreed upon recommendations, and demonstrate benefits. Provide warranty support for impacted components.
Identify areas of performance improvement using performance tools, history reports, and application architecture diagrams
Analysis Data Collection
Analyze usage and runtime data from SMF and RMF to identify the most expensive workloads
Figure 2: Performance Optimization Execution Approach
HCL follows a systematic approach in identifying areas of improvement in mainframe systems and processes. Here is a glimpse of that approach.
In batch jobs, HCL
Examines the critical path in the batch cycle in order to identify undue waiting times Identifies the most expensive batch jobs, programs, utilities, and recurring job failures Analyzes performance data reports for selected jobs
In online transactions, HCL
Identifies the most expensive online transactions and recurring failure events Analyzes performance data reports for selected transactions
Mainframe Performance Management 8
Mainframe Performance Management 9
For identifying design and architecture problems, HCL
Understands the pain points in the application architecture Locates the precise problems in the design of databases and external interfaces Analyzes performance data reports for selected jobs and transactions
For identifying specific problems in the database, HCL
Examines contentious issues Considers or reconsider data compression Considers if appropriate Indexes are being used by the query Takes into account the possibility of reorganizing table spaces Explores if partitioning and segmentation of table spaces would be appropriate
For isolating problems at the system and subsystem levels, HCL
Analyzes performance data reports for z/OS, CICS, DB2 and IMS systems Analyzes all system and subsystem parameters Examines Buffer Pool, Sort Pool, EDM Pool, and DB2 Catalog
PERFORMANCE MANAGEMENT BEST PRACTICES
HCL uses a set of performance management best practices that enable it to:
Identify early signs of performance problems by using performance tools and system related performance reports.
Automate real‐time problem detection and resolution by putting in place performance indicators and thresholds.
Prevent the occurrence of problems by using the SMF and RMF data reports to allocate workload resources based on performance. Historically, the stellar reputation earned by mainframes (for executing high performance operations continuously) has come, in part, from the fact that engineers have been able to allocate system resources proactively. This proactive allocation was not a problem when mainframe systems were siloed, software changes were relatively infrequent, and workloads were predictable.
Benefits The overall business benefits that clients have enjoyed from an improvement in the performance of mainframes are lower TCO and increased productivity. However, as illustrated in Table 1 and Table 2, there are other specific advantages as well.
US‐based Auto & Home Insurance Company
Area of Engagement
Performance Optimization of applications in scope
Solution
SQL query tuning ‐ overall 60% gain in performance
Response time improvement ‐ overall 73% faster views
New indexes to improve performance
Data model redesign improvements
WLM goal mode policy recommendations ‐ performance gain of around 20%
All small jobs now run under a minute as compared to 5 – 60 minutes earlier
Customer Benefits
Reduced TCO
Enhanced cost effectiveness
Reduced technology risk
Optimized business functions and their mapping to system landscape
Improved global visibility of applications and projects
Reduced time to take decisions for new initiatives
Business Objectives Improve the performance of the product in production and recommend improvements in consumption, response time, database improvements, architecture, WLM, etc before its Phase 2 development.
Technology Mainframe Technology (REXX, DB2 V7, ISPF Services, JCL, OMEGAMON for DB2 & other technologies)
Engagement Model/Size Onsite (30%) – Offshore (70%)
Project Duration 4 months
Table 1: Performance Optimization – US Client
Mainframe Performance Management 10
German Financial Services Provider with Global Operations
Area of Engagement
Performance Optimization of applications in scope
Solution
Expected Optimization
Addressed CPU consumption – ~ 30,000 CPU Hours
Expected optimization for HCLTs recommendations – Average ~ 15%
Optimization Recommendations
SQL query tuning (100+ recommendations)
New and modified Indexes to improve performance
Application code performance improvements
Architecture & application redesign
Customer Benefits
Reduced TCO
Easily manageable
Define a formal IT governance
Enhanced cost effectiveness
Reduced Technology risk
Optimized business functions and their mapping to system landscape
Improved global visibility of applications and projects
Reduced time to take decisions for new initiatives
Business Objectives Provide performance optimization recommendations for applications, SQL tuning, related sub system and related areas
Technology Mainframe Technology (COBOL, DB2, CICS, JCL, DB2 PM, OMEGAMON for DB2 & other technologies)
Engagement Model/Size Onsite (10%) – Offshore (90%)
Project Duration 4 months
Table 2: Performance Optimization – German Client
Mainframe Performance Management 11
Mainframe Performance Management 12
Endnotes 1 One of the 50 questions that the research team asked 482 senior financial executives was: who did IT report to in their organization? The response was as follows: 42% reported to the CFO, 33% to the CEO, 16% to the COO, 2% to a chief administrative officer and 7% to other officers. For full details of the survey results and analysis, see John E. Van Decker, “2010 Gartner FEI Technology Study: The CFO as Technology Influencer" (9 April 2010 ID Number: G00175773) p. 2 http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&id=1338722&ref=clientFriendlyUrl [October 2010]‐>represents when an URL was accessed.
2 Note: a discussion on the pros and cons of mainframe lies outside the scope of this paper.
3 IBM’s share of the market for mainframe is believed to be around 90%. Other mainframe vendors include Computer Associates, BEA, Microsoft, Hewlett‐Packard, Fujitsu Siemens, Unisys, Sun, and Micro Focus.
4 The latest being antitrust complaints by TurboHercules, a French maker of open‐source software for mainframe computers, and T3 Technologies, an American distributor of Flex software that runs mainframes files. On the basis of those complaints, the European Commission has opened an investigation against IBM. For details, see Kevin J. O'Brien, "Europe to Investigate Antitrust Complaints over I.B.M. Mainframes" (July 26, 2010) http://www.nytimes.com/2010/07/27/business/27blue.html [October 2010]. See also "Antitrust: Commission Initiates Formal Investigations against IBM in Two Cases of Suspected Abuse of Dominant Market Position" (July 26, 2010; Ref.: IP/10/1006) http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1006&format=HTML&aged=0&language=EN&guiLanguage=en [October 2010]. See also "NEON Files Motion for Partial Summary Judgment in NEON v. IBM" (September 15, 2010) http://www.marketwatch.com/story/neon‐files‐motion‐for‐partial‐summary‐judgment‐in‐neon‐v‐ibm‐2010‐09‐15?reflink=MW_news_stmp [October 2010]. Similar concerns have been expressed in a comprehensive study by the Indian Council for Research and International Economic Relations. For details, see Rajat Kathuria, et al., "The Issues of Competition in Mainframe and Associated Services in India" (March 11, 2010) http://openmainframe.org/featured‐articles/the‐issues‐of‐competition‐in‐mainframe‐and‐associated‐servic.html [October 2010].
5 Kevin J. O'Brien, Ibid. According to Toni Sacconaghi of Bernstein Research, nearly 40% of IBM’s profits are mainframe‐related. Cited in “The Return of the mainframe Back in Fashion” (January 14, 2010) http://www.economist.com/node/15276714?story_id=15276714&source=hptextfeature [October 2010].
6 He is referring in particular to the movement toward client/server platforms and complementary metal‐oxide semiconductor (CMOS) technology combined with the availability, reliability, security, scalability, and manageability of mainframes, the growth of the Internet and e‐business, and the Y2K problem, all of which led many to believe that the mainframe was living on borrowed time. For details, see John R. Phelps, “The Future of Mainframes Looks Surprisingly Good,” (August/September 2003 issue of zJournal, now a part of mainframezone) http://www.mainframezone.com/it‐management/gartner‐cio‐update‐future‐of‐theibm‐mainframe‐the‐looks‐surprisingly‐good [October 2010].
Mainframe Performance Management 13
7 According to IDC, the worldwide server revenue in 2009 was said to have declined 18.9% to $43.2 billion when compared to 2008, and worldwide unit shipments declined 18.6% to 6.6 million units over the same period. For details, see IDC, "Worldwide Server Market Rebounds Sharply in Fourth Quarter as Demand for Blades and x86 Systems Leads the Way" (24 Feb 2010) http://www.idc.com/getdoc.jsp?containerId=prUS22224510 [October 2010]. According to one estimate, IBM’s mainframe revenue took a plunge of 39% in the second quarter of 2009. See Steve Hamm, "IBM Defends Its Big Iron" (August 4, 2009) http://www.businessweek.com/technology/content/aug2009/tc2009084_001429.htm [October 2010].
8 There are several articles on this subjects. See, for instance, Zacks Investment Research, "IBM’s Mainframe Wins Customer" (June 30, 2010) http://www.zacks.com/stock/news/36260/Zacks+Analyst+Blog+Highlights%3A+Union+Pacific%2C+IBM%2C+Hewlett+Packard%2C+Vornado+Realty+and+KKR+Financial [October 2010]. See also Robert L. Mitchell, ""Morphing the mainframe" (February 6, 2006) http://features.techworld.com/operating‐systems/2229/morphing‐the‐mainframe/ [October 2010].
9 The zEnterprise systems is said to start at about $1 million and according to Lou Miscioscia, an analyst with Collins Stewart, they are expected to generate a profit margin of about 70 percent, vs. a 46 percent margin for the company as a whole. Cited in By Katie Hoffman, "IBM Mainframes: Boring but Profitable" (July 22, 2010) http://www.businessweek.com/magazine/content/10_31/b4189041885991.htm [October 2010]. See also IBM, "IBM zEnterprise System," http://www‐03.ibm.com/systems/z/hardware/zenterprise/ and also "Success Stories," http://www‐03.ibm.com/systems/z/success/index.html [October 2010]. According to Martin Kennedy, the managing director of Citigroup's enterprise systems infrastructure, "we will be able to collapse multiple existing large systems into the new water‐cooled z196, which is a huge step in the company's attempt to shift more of its IT dollars away from internal operations and maintenance and toward customer‐facing efforts. For more details, see Bob Evans, "Global CIO: IBM's Blazing New Mainframe Wins Raves from Citigroup," (September 2, 2010) http://www.informationweek.com/news/global‐cio/interviews/showArticle.jhtml?articleID=227200199&pgno=2&queryText=&isPrev= [October 2010]. See also John R. Phelps and Mike Chuba, "IBM Adds Integrated Management Role for Mainframe," July 27, 2010; ID: G00205553) http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&resId=1411013&ref=QuickSearch&sthkw=mainframe [October 2010].
10 MIPS reflect the mainframe’s footprint, and over the past decade it has witnessed a steady rise with installed capacity reaching a high of 11.1 million in 2007. For details, see Timothy Prickett Morgan, "The IBM Mainframe Base: Alive and Kicking" (July 10, 2007) http://www.itjungle.com/big/big071007‐story01.html [October 2007].
11 For a quick overview on this aspect, see IBM, "The IBM Enterprise System Advantage ‐ Running the World's Most Sophisticated Business Transactions," http://www‐03.ibm.com/systems/migratetoibm/getthefacts/market.html [October 2010]. See also Anne Rawland Gabriel, "Debate over the Future of Mainframe Computing Rages On" (September 16, 2008) http://www.wallstreetandtech.com/it‐infrastructure/210601591?pgno=1 [October 2010].
12 For instance, in the future banks need to accommodate for the increasing number of customers accessing mobile banking application. According to a study by Berg Insight, "the number of active users of mobile banking and related financial services worldwide is forecasted to increase from 20 million in 2008 to 913 million in 2014."
Mainframe Performance Management 14
Berg Insight, "Mobile Banking and Payments," p. 3, http://www.berginsight.com/ReportPDF/ProductSheet/bi‐mbp‐ps.pdf [September 2010]. Anticipating such high volume, IBM has developed a 5.2GHz chip ‐ considered to be the highest speed rating to date ‐ for its fastest mainframe computers. See Brooke Crothers, "IBM Ships 5.2GHz Chip, Its Fastest Yet" September 1, 2010) http://news.cnet.com/8301‐13924_3‐20015297‐64.html#ixzz10uOqdeX8 [September 2010].
13 Ryan Arsenault’s observation captures the views of many experts thus: “[t]he problem is not the mainframe platform, but the fact that many insurance companies are making use of legacy mainframe application code which was originally designed to provide quotations to real people.” What is needed is an improvement in the performance of hardware and software applications that use computer resources for running a transaction such as generating insurance quotes. For details, see Ryan Arsenault, "Mainframes Run into Performance Problems with Online Insurance Comparison Shopping," http://itknowledgeexchange.techtarget.com/mainframe‐blog/mainframes‐run‐into‐performance‐problems‐with‐online‐insurance‐comparison‐shopping/ [September 2010.
14 Improving performance of business applications is a key to delivering business value to customers and their end‐users. A recent survey by the Aberdeen Group found that companies that monitored and measured application performance at the point of consumption by end users had few end user complaints. For full details of the survey, see Jeffrey Hill, “End‐User Experience Monitoring and Management” (August 2010) http://v1.aberdeen.com/launch/report/benchmark/6778‐RA‐end‐user‐experience‐monitoring.asp [October 2010]. Writing about the excessive significance given to operational performance metrics in the post 1990s era, Michael Bitterman noted that "IT continues to focus on operations and...explain ‘how the clock works’ when users are asking, ‘what time is it?’ The availability of tools for monitoring hardware and network performance [has given] IT the ability to overwhelm users with statistics regarding IT performance. Users [become] baffled by the metrics that show that mainframe availability was 99.99% yet they couldn’t enter data or access systems.” For the complete report, see Michael Bitterman "IT Metrics for the Information Age" (July 25, 2004) http://www.performance‐measurement.net/news‐detail.asp?nID=198 [October 2010].