Dw Quest+Oct+2005

Business Intelligence and Data Warehousing Group- Satyam Computer Services Ltd.

From The Editor’s desk

Greetings! Here’s another information packed edition of DW Quest! In this edition we present you with an informative paper that gives direction on process-centric approach to business decision-making. This paper promises to give insight on improving capability of a business to gain insight and manage performance. The paper on model management deals with the art of managing various business models associated with project life cycles. Read on to know why a project needs models and what are the various models available. Tuning information systems has been a time honored system management activity ever since the first information system was built. The article presented talks about this concept that helps in enhancing performance. Happy Reading and here’s wishing all our readers a very Happy Diwali! Do mail in your articles to Quest_BIDW !

Editorial Board Anita Shelton Bala Seshadri Hrishikesh Gokhale Manasa Subbaraya Rajasekhar Nanduri Ruy Fregoso Vinod K Srikanth V C

CONTENTS Ø Business

Intelligence to Business Performance Management PAGE 1-6

Ø Model

Management PAGE 7-15

Ø Datawarehous

Tuning PAGE 16-20

Satyam Computer Services Ltd. (NYSE: “SAY”) is a leading global consulting and IT services company that offers a wide array of solutions for a range of key verticals and horizontals. Business Intelligence and Data Warehousing is a focused competency unit whose mission is to enable customers to take better decisions by providing world-class BI solutions. For more information please refer to www.satyam.com.

Disclaimer The views expressed in this document are not those of Satyam Computer Services Ltd, and any errors or omissions in the contents are attributable to the respective contributors / copyright holder. Customer names are withheld to ensure confidentiality. © All rights Reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the publisher.

CONTACT INFORMATION Brian C. Coffey Ramakrishnan TS SanjayKumar Joshi Arun VS Sarvesh Kumar 1000 Windward Concourse Suite 540 Satyam City Centre 1, Gatehall Drive 360 Elizabeth Street 6 Cedarwood Alpharetta, Georgia 1-10-60 to 63, Satyam City Centre Suite 301, Parsippany Level 40, Melbourne Central, Chineham Business park USA 30005 Begumpet, Hyderabad-500 016 NJ 07054, USA Melbourne, VIC, Australia Basingstoke, Hampshire (W) 770-442-3111x2201 (W) +91-40-5523 2323 (W) +973-656-0650 (W) +61 3 86605600 England-RG248WD (M) 404-435-0472 (M) 91-9849830753 (M) 518 248 7887 (M) 614 1616 3136 (W) +44-0-1256-394100 (M) 0044 791 933 9137 Kiran Cavale Sastry Varanasi Victor Samuel Indira Munjuluri Venkat Narayanan #97, Galaxy Towers, Satyam City Centre #97, Galaxy Towers, 251-252, Solitaire Knowledge Dynamics-A Div of BI G.N.Chetty Road 1-10-60 to 63, Satyam City Centre G.N.Chetty Road Corporate Park, Mumbai (W) : (65) 6324 6244 x 116 T.Nagar, Chennai-6000 17 Begumpet, Hyderabad-500 016 T.Nagar, Chennai-6000 17 Maharashtra 40 0093, Fax : (65) 6324 4355 (W) 91-44-2826120 (W) +91-40-5523 2323 (W) 91-44-2826120 (W) +91-22-5695 0166 M: (65) 9682 1535 (Singapore) (M) 9841080505 (M) 91-9849496082 (M) 9841027947 (M) 91-9821723010 M: (91) 9840702328 (India)

http://www.satyam.com

Business Intelligence and Data Warehousing Group- Satyam Computer Services Ltd. 1

Business Intelligence to Business Performance Management (Gayatri Jayanthi)

Abstract

In the face of increasing business environment, Business Intelligence alone is in-sufficient to manage the performance of an enterprise. A new methodology is required to build, communicate and execute better strategic plans. Business Performance Management (BPM) is a new term that denotes a process-centric approach to business decision-making. This is intended to improve the capability of a business to gain insight and manage its performance at all levels by combining stakeholders, managers, staff, customers and suppliers within an integrated management environment. It combines the management methodologies like Six-Sigma with business requirements for executive reporting and budgeting.

Some of the components that are embedded in Business Performance Management are

• OLAP – Online Analytical Processing

• Query and Reporting

• ETL

• Analytics

• Budging

• Consolidation

• Metrics

• Portal dashboards

• Scorecards Challenges faced by Business

1. Attaining Strategic Goals: In any organization, there must be a shared commitment towards strategic goals. If there is a divergence observed in this, then there are chances that resources will be mis-utilized. Hence it is BPM which needs to drive strategic focus towards the same goals by broadcasting that local goals are congruent to the Corporate Objectives.

2. Optimizing Communication between Organization Units: As the organization grows in size, it is a huge challenge to have a cross communication among the various business units and interaction between their management process. Every unit is focused on their local goals and targets


rather than attaining the organizational goals. BPM enables collaborative management by providing cross departmental data under a single umbrella.

3. Decision Making: Apart from the timely availability of data, the biggest

question that a manager needs to answer is how far the data is accurate and consistent as the decision made on this data is like working in darkness with out a clue of the under lying facts.

BPM provides a fact based decision that is based on controlled sources.

4. Knowledge Sharing: It is a known fact that even today many of the

information analysis is done with the help of spread sheets. This information gets locked with the person who writes it. This is specially observed during Budgeting and forecasting where the data gets locked in spreadsheets unavailable in the primary systems. BPM looks forward in evolving a complete transparency by documenting all forms of management information.

5. Accessing other Operational systems: In addition to all the above

challenges, interfacing other Operational Business systems such as Supply Chain Management, Enterprise Resource Planning, Employee Relationship Management and Risk Management is also essential as they play a major in all executing the strategic plans.

6. Adapting to Business Demands: Responding in a quicker way to market

demands is also seen as a major challenge.

Need for BPM

It allows managers across the extended supply-chain, to collectively measure performance, anticipate results and drive profitability. It brings together loosely coupled business units of an organization and moves forward meeting organizational goals. Performance Management is just one of the management tools which can be used to measure what your business is doing and also help to align the activities of people and processes to achieve the desired goals.

• BPM provides an infrastructure to have a peer – to – peer collaboration between Business Units for facing the changing Business demands with out losing control over the business.

• Strategies like outsourcing, shared services lead to a new organizational model which is open and fluid. BPM helps managing the Business Processes by protecting sensitive information of each participant organization.

• Many huge organizations have deployed Enterprise Business Applications such as ERP, Supply Chain Management to increase the efficiency of the Business by streamlining the operational processes and generating valuable data. But many a times BI on these ERPs focus either on summarized data or provide massive information which may not be the demand of operational managers. BPM


absorbs the operational data to the granularity that makes sense for the monitoring and analysis management activities.

This paper will focus on the need for moving towards BPM, challenges in implementing BPM and also how the increase in BPM implementation would influence service providing companies.

Introduction

Data on its own does not have a meaning unless it is interpreted to make it information. Data will turn to be informative when used for analysis or reasoning. Any organization which empowers itself to utilize its information in its decision-making and know its strengths and weakness will head towards success. As organizations realized the importance of information, they turned away from decisions made on intuition to a fact-based analysis. This lead to the evolution of Business Intelligence, organizations started automating more systems and hence more data is available. However collection of all this data in a single repository was a challenge which was answered by Data warehousing technologies. Business Intelligence is now the art of extracting large amounts of data, turning it into information and presenting it in an actionable knowledge.

With the advent of Business Intelligence, organizations felt the demand of aligning their action with strategy to improve its performance. Thus came BPM as it is recognized as a way to align and optimize its performance such as sales, marketing, customer service, IT, compliance, Finance and HR departments across an enterprise. BPM is seen as the next generation of Business Intelligence (BI). BPM is focused on business processes such as planning and forecasting. It helps businesses discover efficient use of their business units, financial, Human, and material resources. BPM often uses Key Performance Indicators (KPI’s) to indicate the present state of business and to prescribe the course of action.

According to Gartner "Corporate Performance Management (CPM) or Business Performance Management is an umbrella term that describes the methodologies, metrics, processes and systems used to monitor and manage the business performance of an enterprise." While less than 10% of Global organizations had implemented Business Performance Management at the end of 2002, Gartner predicts that more than 40% will do so by 2006.

What is Business Performance Management?

BPM is a combination of 5 components 1. Participants: BPM is about how people come together in order to drive the

performance of the organization. There are many business activities namely manufacturing products, fulfilling orders etc which must be controlled and


organized by management activities. BPM identifies different roles that any individual can play with regard to a particular management activity

• Stakeholders • Suppliers • Partners • Influencers • Managed Staff • Managers

BPM identifies all these roles and presents a complete view of the management process.

2. Management Process: This represents the on going dynamic state of the business. The management teams must interact with their stakeholders, customers, suppliers and staff as well as other managers in different teams for developing plans to meet their organizational goals.

3. Management Practices: BPM is a mechanism for recording or discovering

the existing definition, processes, metrics and business rules of the management. The management practices defines few management activities like negotiating goals with the stake holders, developing and communicating plans, variances between prediction and actual and informing Stake Holders. Apart from this, activity management practices, also define KPI’s that measure key business drivers.

4. Conceptual Framework: It is a bridge between the management practices

and the under lying systems. BPM defines a universal framework describing processes, activities, information, business rules and user roles. Without this Framework it would be extremely difficult to have the same Management Practices supported by different systems. This is the case as systems are replaced over time or when different business units are acquired and have their own systems already in place.

5. Systems and Technology: This is one of the most important component of BPM. It enables Management Process, Practices and Framework to the participants of the system. This provides a personalized view as per the role. There will be a set of tools and applications supporting this underlying system in order to make the dashboard or scorecards available such as OLAP – Online Analytical Processing, Query, Reporting, ETL, Analytics, Budgeting, Consolidation, Metrics, Portal Dashboards and Scorecards etc. The combined network of all these tools and applications enables the Business managers to explore innovative alternatives and share them with their colleagues. The figure below shows the systems, their technology and their inter dependence to make BPM.


The deployment of Business Performance Management across the enterprise, helps to communicate with the operational systems, directly or via an intermediate data warehouse layer. The figure below shows the pictorial depiction of the components of BPM

BUSINESS MANAGEMENT

PARTICIPANTS

INTERACTIVE MANAGEMENT PROCESS INSIGHT ACTION

MANAGEMENT PRACTICES

CONCEPTUAL TEXT

SYSTEMS AND TECHNOLOGY


Conclusion As the business environment becomes highly volatile and fast moving, businesses tend to face a lost of inefficiencies in management and process. These inefficiencies risk the quality of management processes. Organizations face strategic misalignment, unreliable decision making, hidden knowledge and non-adaptability to changing business demands. This situation is being faced by many of the big organizations as business is purely focused on management solutions that are based on single activity tools like Budgeting, reporting etc. BPM is a strategic tool, which can improve management processes by providing fact based decision making, strategic focus, collaborative management and thereby increase their return on investment. References

1. Business Performance Management – Hyperion By Alistair Shaw 2. Article published in DM Direct Special Report – Feb 24,2004, By Greg

Mancuso and Al Moreno Authors Profile Gayatri Jayanthi is a Senior Software Engineer in the Data ware Housing and Business Intelligence Practice. She has an over all experience of 5 years, out of which last 4 years have been in Data ware housing exclusively on Informatica, Business Objects and Brio. Currently she is leading a team working on Data Stage Parallel Extender. Email Id : [email protected]

mailto:[email protected]


Model Management (Keshav Tripathy)

Abstract Nowadays all IT organizations have to execute projects within challenging conditions: Pressure on price, time, maintaining and improving the quality, managing the resources and taking care of unforeseen risks. Hence, it becomes imperative to be prepared for the variations of the above parameters before hand. This makes it necessary to plan and execute the project on specific guidelines, taking the variations of the parameters into consideration. To do so, managing the various models associated with a project life cycle becomes crucial. Models are representations of Functions and Information in use or planned for use by any organization. Models may be of Business Oriented which reflect the business activities and information or may be Information System oriented and reflect the functions and data of a specific information system. The models are developed keeping in mind the purpose of the project and the parameters or variables the project is dependent on. The models are developed based on specific modeling techniques and have a firm mathematical background to it. Effective Model Management means ensuring that the correct model is available to individuals within the business and the Information System groups as needed. This call for making models available at begin of projects, incorporating changes and providing models to the business community. It may also involve coordination of model usage across projects and the validation that they reflect the business as it is and that they conform to the Organization Standards. This paper does not do is describe the various models for project management but it discusses the model management as one of the integrated functions of project managers. It introduces what are models, why a project needs models and what are the various models available. The paper discusses on the importance and usage of model management for projects by other stakeholders of the project viz. the team and the business groups. As many people may interact with the model with various roles as project manager, team member, customer etc, it becomes necessary to lay out rules on the management of the model so that it’s integrity is maintained. Overall the paper introduces to the reader the need for model management in software development project and other issues associated with it.

1. Introduction

Creating a business solution on time and within budget requires a disciplined approach. Most large IT organizations rely on development methodology to keep projects on track, including some developed during the mainframe era. These older methodologies are often too inflexible to provide value in today's distributed Internet. But whatsoever it may be the solution space a robust Project Management Methodology based on Modeling Technique is essential to make the Project Success. 2. What is Project Management?


Simply put, it's a way of managing complex tasks. In reality, however, project management is both an art and a science in which a specific regimen of principles, tools and techniques is combined with political savvy, good negotiation and people skills.

Every company, large and small, can benefit from good project management skills. Incorporating project management into the work environment leads to better efficiency and increased profitability. It helps a company consistently achieve goals and objectives. It enables a small company to compete by making better use of limited resources. It can help a company to execute projects on time, within budget and with consistently high quality. Project management is a proactive way to prevent fires. In today's tough business environment, good project management based on strong models can give a company the competitive edge.

3. What are Models Models are representations of Functions and Information in use or planned for use by any organization. Models may be of Business Oriented which reflect the business activities and information or may be Information System oriented and reflect the functions and data of a specific information system.

Normally these models are diagrams that facilitate the understanding. Such diagrams can illustrate complex processes, such as the Flow of Information through the business, or a simple process like as a List of activities or data groupings. The models may be developed in many different environments and contexts: an Information System development project, a new manufacturing system; early models to show the major functions of the system; later in project development to illustrate the detailed processing logic. The business may also use modeling as part of a business analysis effort. 4. Why Does Management Need Models? There are three primary reasons why project management needs models associated with a systems development project:

1. To estimate the money, time, and people required to develop the project. 2. To update and revise those estimates as the project continues. 3. To manage the tasks and activities of the people working on the project.

Budget and estimates are one of the major activities of managers in any project; they are all the more difficult in many systems development projects because each project is (or at least seems to be) unique. However, management still needs models — graphical models are clearer — to see the availability of the people, what will happen if those people are not available, and so forth. Even the best plan, though, is likely to fail if it is implemented blindly. Circumstances change continuously during the project: critical resources not available; inaccurate


work estimates; sudden changes on the user’s dates; work completed more slowly than originally expected. Thus, it is important for the project manager to have models that can let her or him explore the consequences of changes in the plan.

And finally, the project manager must not only manage tasks, he or she must manage people. The manager must ensure that all the personnel are doing what they should be doing, when they should be doing it. The manager needs modeling tools that will focus on the people, in addition to modeling tools that focus on the tasks 5. Purpose of Models 5.1. Documentation: A primary purpose of models is to document the business and the system that support the business. Such documentation, especially in graphical form, can provide valuable information to business on how it operates.

5.2. Understanding the Business: Through the development of Business-Oriented Models, allow a better of the business functions and then information used to perform those functions. Understand the business problem should be the foundation of any Information Systems work performed. and the models illustrate to the Information Systems group how the systems support critical business functions. 5.3. Understanding an Information System: System Oriented Models help the business and the Information Systems group better understand a specific set of programs and databases. Visually they can identify major system functions, such as Planning and Inventory Control within a New Manufacturing system, as well as detailed system processing, such as the algorithm used for Production Planning. These models also help explain

5.4. Communication Tool between the Business and Information System: By developing common business oriented terminology communication barriers can be reduced between a specific business unit and the information systems group that supports it. The graphical nature of many models also assists communication: It may require several pages of text to describe how Order Information flows from Order Entry System to the Order processing System to Order Fulfillment System but a simple one-page diagram may illustrate this same information in a better way.

5.5. Improving System Quality: From the System developer’s perspective, models can improve the quality of systems analysis and design. For example when looking at new H.R. System from this high level it may be easy to identify that, while hiring and terminating employees can be supported, there is no mechanism for transferring employees. At the design level the developer can define detailed processing logic graphically which will facilitate the identification of omissions and inaccuracies by the developer and the other team members.


5.6. Improving Business Processes: Businesses are using models of their business processes as the foundation for improving process efficiency, by documenting current activities and modeling new approaches, the business can easily identify current inefficiencies and possible solutions. This has applicability to activity improvement, job redefinition and departmental reorganization.

5.7. Project Team Coordination: Within an Information System project team, models can become a means of coordination. They can depict a high level overview of how a system will function so that all team members can understand how their area of responsibility meshes with the entire system. A more detailed model can help the team members understand specific relationships between their activities and those of then other team members.

5.8. Cross Team Coordination: Many of the benefits available with a Project Team approach are also available across the Information Systems Department. Specially in integrated system environments, it is critical that analysts understand the way in which their system integrates with other systems. This understanding facilitates better design and maintenance of integrative process and data.

6. Project Management Models Project Management Models which are very successful in executing complex projects have been identified as key to success of Modern Project Management. Some of these are the following. • Enterprise Architecture Model: Designed to shorten the enterprise

architecture planning cycle, this model provides guidelines for building enterprise architecture through versioned releases.

• Team Model: Designed to improve internal teamwork, this model provides a flexible structure for organizing teams by project. The Team Model can be scaled to meet a project's scope, a team's size, and the team members' skills.

• Process Model: Designed to improve project control, minimize risk, improve quality, and shorten delivery time, this model provides project structure and guidance throughout the project life cycle. The Process model describes the phases, milestones, activities, and deliverables of a project, and explains their relationship to the Team Model.

• Risk Management Model: Designed to help the team identify priorities, make well-informed and strategic decisions, and prepare for contingencies, this model provides a structured and proactive environment of decisions and actions to continuously assess, prioritize, and deal with project risks.

• Design Process Model: Designed to help discern the true business and user needs behind a technology project, this model employs a user-centric continuum to enable efficient and flexible design through a parallel and iterative approach. The conceptual, logical, and physical design phases provide three different perspectives for the three different audiences: users, team, and developers.

• Application Model: Designed to improve development, maintenance, and support, this model provides a three-tiered, services-based approach to designing


and developing. The services utilized in this model are scaleable, and can be used on a single desktop or on multiple servers worldwide.

These models promote principles such as risk-driven scheduling, a fixed-release-date mind-set, versioned releases, visible milestones, and small, peer-based teams. Modern Project Management exposes critical risks, important planning assumptions, and key interdependencies required for successfully planning, building, deploying, and managing the company's mission-critical technology infrastructure or business solution.

7. Why Model Management In any organization small isolated groups start developing project specific models. Gradually more groups begin to develop project specific models and then people start to edit some other person’s model. At this point the organization needs some form of Model Management.

Effective Model Management means ensuring that the correct model is available to individuals within the business and the Information System groups as needed. This call for making models available at begin of projects, incorporating changes and providing models to the business community. It may also involve coordination of model usage across projects and the validation that they reflect the business as it is and that they conform to the Organization Standards.

8. Challenges of Model Management If you see from the surface it seems like a relatively simple task as because it is the one that information system personnel have been dealing with success for several years in the area of program source management (i.e. Source Control or Version Control), Data Dictionary Administration etc. However a Model Manager must deal with some unique problems. • They must manage many different types of objects with complex relationships.

The model management deals with Program Sources, Data Dictionary and also manages all the process-related objects, possible planning objects and the relationship between the objects.

• Unlike Data Objects and Dictionary and Programs, most model objects have limited standardization. Different vendors have come up with separate models in a different way. Each methodology may dictate the use of a different set of modeling objects or different object relationship. Same models can be depicted in different ways. Therefore determining what constitute a real change to a model is difficult.

9. Types of Model Management • Project Model Management: The Project Model Management involves in

tracking the development of models for a specific projects. This type of Model


Management is restricted in scope. The challenge here is to ensure that valid models are available to all Team Members.

• Corporate Model Management: This is to manage the models across all Projects in the Organization. It also involves management of Models that are not currently a part of the project but are of interest to the Organization.

The given Figure 1 below describes Project and Corporate Model Management as separate activities.

10. Functions of Model Management 10.1 Establishing and Enforcing Standards One of the primary functions of model management is the establishment and enforcement of model standards. To develop these standards the model manager must first determine what types of objects are required and what types are optional for the models. Required and optional relationships between objects should also be identified. For each of these objects the model manager must identify the information that should be collected. A critical part of this detailed information is the name of the object. For objects that represent the physical pieces of the system, the physical naming standards can be used. New naming standards must be developed for other objects.

10.2 Establishing the Organization’s Information Model An approach to documenting object-related standards is through an information model. It is a diagram that shows the required and optional objects and relationships for the organization. A more detailed model may also include specific properties such as name and definition that are also tracked about an object. Figure 2 below shows a sample information Model. This is a high-level information model.

Fig: 1

Fig: 1


10.3 Establishing and Enforcing Procedures

Once the standards are determined, model managers need to identify how the standards will be enforced. Procedures identify how and when models will be updated. It is also helpful to identify clearly who is responsible for performing the updates. If corporate models are being developed it is also important to determine how project models will be merged together to form corporate models. In establishing model maintenance procedures the model manager needs to identify all possible points where models can be changed. Information System Development Projects are one obvious source of change. Business changes, system maintenance and planning projects are also possible source of change. In each of these scenarios the model manager needs to identify how the model will be changed and who will perform the update.

10.4 Establishing the Organization’s Activity Model Model management procedures can also be documented through a model. While the information model documents the types of information to be managed, the activity model documents the steps for managing those models. An activity model may be at a very high level identifying the major functions to be performed or at a very detailed level documenting the procedures themselves. Figure 3 & 4 below depict a sample activity Model. First one depicts a Process Model decomposition diagram format. The second one illustrates which portion of the information model is affected by each activity.

Fig: 2


10.5 Tracking Model Updates Both the standards and procedures should include a mechanism that helps track model maintenance. Tracking involves two activities: identifying that a change has occurred and reacting appropriately to the change. Model managers must ensure that all updates that result from the changes are completed. Once a change has been identified, the model manager should also ensure that all associated changes have occurred. EXAMPLE: If a business process has been changed, the programs that support that business process should also be changed accordingly. Although the model manager

Fig: 4

Fig: 3


is not responsible for making such changes but he should inform the individuals who maintain these programs. This requires tracking of not only the objects themselves but also the linkages between objects and the individuals who have the responsibility of maintenance.

11. Project Level Model Management

The most basic form of model management occurs within a project team. On a team with one individual, the developer is responsible for ensuring that all required objects are created and are linked appropriately to provide them information necessary for efficient and effective application development. However, when more than one individual is involved in the model development activities for the project, model management becomes more complex task. The work of all developers must be coordinated to ensure that the resulting models accurately combine to represent both the business and the system that support it. Critical issues in project level model management include assigning responsibility for objects and linkages to the team members, coordinating the use of shared objects, and resolving conflicts within the model. It is also important to identify clearly for the team which objects and linkages are required. The team can then identify which diagrams and modeling techniques will best help them meet model management and project communication requirements.

12. Conclusion We have seen that how crucial is the function of a Model Manager in a Project and Model Management activity must be aligned with the Project Management activity. Every Project Managers must take into this consideration for managing large projects which are more of developmental in nature. By ensuring proper model management in a Project one can ensure integrity of the models across all the phases of the project as well as it will provide seamless integration with different modules of the application systems. 13. References

1. Software Project Management: A Unified Framework: Walker Royce, Addison Wesley

2. Software Engineering Economics: Boehm, Barry W: Prentice Hall Englewood Cliffs, NJ 1981

3. Industrial Strength Management Strategies: Brown Norm, IEEE Software V-13 N-4 July 1996

4. Edward Yourdon’s Modeling Tools for Project Management. 5. Database Modeling and Design: Tody J Teorey, Morgan Kaufman Publication

Author Profile Keshav Tripathy – Has been with Satyam from February 2001. Currently he is working with EABIS (BI &DW)Circle, Bhubaneswar.. [email protected]

mailto:[email protected]


Data Warehouse Tuning Whenever a system matures and performance starts to become an issue, there is an activity known as “Tuning the system” that assumes importance. When a system is tuned, it yields better performance without any major and wholesale changes to the system. Tuning the system has been a time honored system management activity ever since the first information system was built.

What is Data Warehouse Tuning?

Tuning the Data Warehouse / DSS environment – for the purpose of this article – is taken to mean the activities that will enhance performance that stop short of a wholesale rewrite and reconstruction of the data warehouse. It also means the removal of the data warehouse off one hardware platform and onto another platform.

Tuning begins with the monitoring of the activity and the data that flows through the data warehouse and the DSS environment. If there is no monitoring, then finding the exact cause of the slow performance will be a difficult task. In order to tune the system the Data Warehouse administrator must know what is going on in the system. Otherwise tuning is merely a guessing exercise.

A monitor indicates the administrator what is going on inside the system. It also tells the DWA how effective the attempts at tuning have been, once the changes are made inside the data warehouse. Therefore, it is essential that both activity and data be monitored before the tuning exercise commences and during the tuning exercise itself.

Operational Database Tuning Vs. Data Warehouse Tuning

Tuning the system takes on a different meaning and importance in the Data Warehouse / DSS environment than in the classical operational environment. The first reason why tuning is different is that the essence of the data warehouse environment is one of iterative development.

Unlike the operational environment where system requirements are known prior to the building of the system, in a data warehouse environment the developer often does not know what the requirements are going to be until the system is built. Only after the first iteration of the data warehouse is up and running does the developer know for sure what is going on inside the system. This uncertainty ensures tuning the data ware house / DSS environment system after the initial construction is absolutely mandatory.

A second tuning difference between the operational environment and the data warehouse / DSS environment is that of the fundamental nature of a transaction. Operational transactions are mission critical and require 2 - 3 seconds response time. Operational transactions require a higher degree of availability and each transaction uses only a small amount of system resources. Data Warehouse/ DSS transactions are not mission critical and often use a vast amount of resources. It is normal for


many types of DSS transactions to have a longer response time. Furthermore, given the fact that DSS transactions are used for long term planning, not for immediate business decisions, the business case for a very fast response time is weak. Therefore, the acute degree of software engineering required of operational transactions is not required for data warehouse / DSS transactions. In this sense, tuning the data warehouse / DSS environment is not quite critical as tuning the operational environment.

Another factor relevant to tuning is the volume of data found in each of the environment. The data warehouse / DSS environment contains 1 - 2 more orders of magnitude of data than the operational environment. This difference in the volume of data requires the DWA to possess sophisticated management skills, of which tuning is an important part.

Tuning Activities

Creation of Indexes

The simplest tuning activity is that of adding indexes to data based on access and analysis.. When used properly, indexes can reduce the system resources required to locate data. However, a trade off is to be made. Once the index is specified, the maintenance that is required every time data is added to the data warehouse must be accounted for. In addition, indexes require their own space. But for data that is accessed frequently down the same path, indexes can save processing resources far beyond their cost.

Creating Data Marts

Creating data marts is a powerful way to enhance performance. With a data mart, the end user specifies and extracts a selective subset of data from the data warehouse that is customized to meet particular departmental needs. With a data mart, the volume of data can be tremendously reduced. Data can be summarized, and customized. All of these factors lead to a much greater performance in the eyes of the end user.

In addition, data is isolated on a machine dedicated to the individual needs of a department. With this isolation, the department can achieve a boost in performance as they are willing to pay for. The cost of the machines used at the data mart level is significantly lower than the cost of the machines used at the data warehouse level. For these reasons, data marts are a popular approach to achieving performance.

Summary Tables

An alternative approach to data marts is to create organization wide summary tables. These summary tables are open and available to any end user on the data warehouse and not just to the user of the data mart. The summary data is much faster to access than the base data that must be recalculated every time the summary is desired. Therefore, the summary data requires space. This is the case with every unit of summary data, there is a process of summarization. This process of summarization becomes “meta process” information that is necessary to describe the basic nature of the summary data itself.


Removing Dormant Data

Dormant data is data that is not accessed but resides in the data warehouse. Removing dormant data greatly enhances the prospects for performance. Dormant data takes up space and gets in the way of serious DSS analysis. If data is not being used, then it has no place in the data warehouse / DSS environment. The effect of dormant data (or the presence of dormant data) is normally not felt until the warehouse is fairly large and mature. In the early days of a data warehouse, almost all data is being used. Even if the data is not being used, the small dimensions of the early data are such that dormant data hides with no ill effects. But as a data warehouse grows larger, the percentage of the data that is not being used increases drastically. When a data warehouse contains more than 50% of dormant data, then it is time to prune the data warehouse of the data that is not being used.

Star Joins / Snowflake Structures

Another very useful technique to enhance performance is to create STAR JOINS. STAR JOINS for their larger counterpart – SNOWFLAKE structures are useful for streamlining processing. In a STAR JOIN there is a FACT table and several DIMENSION tables. The FACT table is pre joined to the DIMENSION tables to accommodate access to the data. While FACT tables can enhance performance, they must be used with caution. The first problem with STAR JOINS is that they require a foreknowledge of how the data is to be used. If there is foreknowledge, then FACT tables and DIMENSION tables make a great deal of sense. But if there is no foreknowledge of how the data will be used, there is no guarantee that the STAR JOIN will accomplish a boost in performance. A second issue is that one group of people may have a single pattern of usage while another group of users may have another pattern of usage. The STAR JOIN can optimize access based on only one pattern of access. For this reason, STAR JOINS are best equipped to optimize the performance of large data marts, where there is a better chance of a homogeneous usage of data.

Data Partitioning

By partitioning the data finely at the database, there is the chance that a higher degree of parallelization can be reached in processing data. For example partitioning tables based on time id for each and every FACT tables enhances the performance of the query by reducing the time taken to locate the data to a great extent.

Defragmentation of The Table spaces

In a typical Data warehouse / DSS environment running on Oracle Database, space management is very crucial in maximizing performance as well as space usage of a database. Over a time period, the database can become fragmented affecting the performance and resources of the database. If any ORA- errors regarding allocation of extents or space is received and the DBA_FREE_SPACE reveals a lot of free space left in that tablespace, then the database on which the warehouse is running is a victim of fragmentation.

When a tablespace is fragmented, there may be a lot of free space in the tablespace, but it is in such small pieces that Oracle cannot use them. When an object needs a next extent, Oracle has to allocate one contiguous extent. If one does not have a chunk of free space that is large enough for the extent, Oracle will return an error. A tablespace may have smaller pieces of free space that may add up to the size of the


next extent or even more, but Oracle cannot divide the next extent of the object into smaller pieces to fit into the free space. De allocation of extents from an object to the free list will cause a tablespace to become fragmented. The more often this movement of space occurs, the more fragmented the tablespace will become. This phenomenon can be more prominent in a data-warehousing environment, where many tables and indexes are being dropped, and reloaded.

There are two types of fragmentation that may be seen in a tablespace.

• A tablespace may have 2 pieces of free space but in between the two, there is an extent belonging to another permanent object. There is no way to combine these two free extents and repair the fragmentation.

• Another type of fragmentation occurs when the tablespace has two pieces of free space that are adjacent to each other, but they are separate. This type of fragmentation can be repaired, by manually combining the tablespaces.

The only way to get rid of fragmentation within an object is to drop and recreate the object. However one can take care of the second problem mentioned above by manually combining the tablespaces.

Separating FARMERS from EXPLORERS

One of the useful classifications of users in the data warehouse / DSS environment is that of explorers and farmers. Explorers are those people who do not know what they want but will know it when they find it. Farmers are those people who regularly access data having a good idea of what they want. Both types of users are valid and are normal in the data warehouse / DSS environment.

Separating FARMERS from EXPLORERS is one of the easiest and most effective ways to enhance performance.

Farmers submit small requests that access a uniform and limited amount of data. Furthermore they also have a regular pattern of access. They almost always find what they are looking for. Explorers do not know what they are looking for and submit requests on a very irregular basis. Explorers look at huge amounts of data and often find nothing.

The pattern of data and system usage created by farmers and explorers is very different. The patterns are eternally in conflict with each other in terms of system utilization.

The Data warehouse administrator can enhance performance for everyone by separating the transactions submitted by each community. The explorer transactions are run after hours and the transactions the farmers submit are run during the day. In doing so, the workload starts to achieve some degree of predictability.

Tuning the Data Warehouse according to its usage

In order to fine tune the system the Data Warehouse administrator must keep track of what exactly is happening in the system. He must be able to answer the following questions based on the day to day tracking of the Data Warehouse


• What data is being used? How often, by whom and When? • What is not being used? Is it ever being used? • Who is using the Data Warehouse? Should they be there? • Who is not using the system and who should be? • What is the most resource – intensive operations on the system? • What is the average response time of queries across the system?

Asking these questions is the first step towards improving the data warehouse. It is the aftermath of a successful data warehouse implementation that there are questions generally asked by people contemplating the next generation data warehouse.

There are different approaches to getting these usage questions answered. Each approach has its own assumptions, tools and methods. The basic intent behind the strategy to monitor, record, analyze and correct is also known as usage tracking. There are three general mechanisms for usage tracking.

• Purchase or build end user tools which record usage and present it for analysis • Sample usage information from the database’s internal views • Intercept, record and analyze (all or selected) network traffic.

Once these questions are answered to some extent the cause of the slow performance can be located and thus subsequent corrective measures can be taken.

Eliminating “HOT SPOTS”

Of course, where data resides upon parallel hardware architecture, identifying and removing “hot spots” can produce great gains in performance. Even when hot spots can be located, there is no guarantee that they will reappear. Yet, being able to identify hot spots in a valuable piece of information to the DWA, charged with producing optimal performance.

SUMMARY

Tuning the Data Warehouse is fundamentally different from tuning an operational environment. There are many techniques to the elimination of data that is not required or the unnecessary processing of data for the purpose of DSS processing. Some of those techniques include:

• Creation of indexes • Creation of Data Marts for different subject areas • Creating SUMMARY tables • Removing Dormant Data • Star Joins / Snowflake structures • Data Partitioning • Defragmentation of the tablespaces • Separating Farmers from Explorers • Tuning the Data Warehouse according to its Usage • Eliminating “Hot Spots”

Article Contributed by Knowledge Dynamics a unit of Satyam BIDW.

Dw Quest+Oct+2005

Documents

Transcript of Dw Quest+Oct+2005