COMP 578 Data Warehouse and Data Warehousing: An Introduction Keith C.C. Chan Department of...
-
date post
19-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of COMP 578 Data Warehouse and Data Warehousing: An Introduction Keith C.C. Chan Department of...
COMP 578Data Warehouse and Data Warehousing:
An Introduction
Keith C.C. Chan
Department of Computing
The Hong Kong Polytechnic University
2
What is A Data Warehouse?
DataWarehouse
HRO
Health
Student
ITs
A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decision-making process.— W. H. Inmon
3
Data Warehousing and Industry
• One of the hottest topic in IS.– Over 90% of larger companies either have a DW or are starting one.
• Warehousing is big business– Old statistics from Megroup.
• $3.5 billion in early 1997• $8 billion in 1998 [Metagroup]• over $200 billion over next 5 years.
– Latest by IDC on DW tools.• $5 billion in 1999.• $16 billion in 2004.
– Latest by IDC on CRM applications• $61 billion in 2001• $148 billion in 2005
4
Data Warehousing and Industry (2)
• A 1996 study of 62 data warehousing projects showed an average return on investment of 321%, with an average payback period of 2.73 years.
• In 2003, some people are skeptical.• WalMart has largest warehouse
– 900-CPU, 2,700 disk, 23 TB Teradata system– ~7TB in warehouse– 40-50GB per day
Why Data Warehouse?Why The Hype?
6
Information vs. Data
• Information is pivotal in today’s business environment. Success is dependent on its early and decisive use. A lack of information is a sure sign for failure. The rapidly changing environment in which business operates demands ever more immediate access to data. (Devlin, 1997)
• Many corporations are actively looking for new technologies that will assist them in becoming more profitable and competitive. Gaining competitive advantage requires that companies accelerate their decision making process so that they can respond quickly to change. One key to this accelerated decision making is having the right information, at the right time, easily accessible (Poe, 1996).
7
The Information Gap• The information gap is a result of:
– Fragmented way in which ISs and supporting DBs have been developed.• One-thing-at-a-time due to constraints on time and resources.• DBs on a variety of hardware and software platforms.• Difficult to locate and use accurate information.
– Most systems developed to support operational processing.• Operational processing (a.k.a. TP) captures, stores and
manipulates data to support daily operations.• Little thought given to the information or analytical tools
needed for decision making.
8
Bridging The Information Gap
• Data warehouses (DW) consolidate and integrate information from many different sources and arrange it in a meaningful format for making accurate business decisions (Martin, 1997a).
• They support complex business decisions through analysis of trends, target marketing, competitive analysis, and so on.
• Data warehousing has evolved to meet these needs without disturbing existing operational processing.
9
What Are The Issues?
• How DW relates to existing operational systems.
• Data architecture appropriate for most DW environments.
• Extracting data from existing operational systems and loading them into a DW.
• Interact with DW using OLAP, data mining and data visualization.
Data Warehouse & Data Warehousing as Solution
11
The Need for Data Warehouses
• Two major factors drive the need for data warehousing in most organizations today:– Business requires an integrated company-wide
view of high-quality information.– The IS department must separate informational
from operational systems in order to dramatically improve performance in managing company data.
12
Need for a Company Wide View
• Data in operational systems typically fragmented and of poor quality.
• Generally distributed on a variety of incompatible HW and SW platforms:– Unix running oracle DBMS– IBM MVS running the DB2 DBMS
• Often necessary to provide a single, corporate view of that information for decision making.
13
Deriving a Single Corporate View
• Develop a profile for each student from:– STUDENT_DATA, STUDENT-EMPLOYEE,
STUDENT_HEALTH
• Some issues to resolve:– Inconsistent key structures: HKID and student name– Synonyms: Student_No and Student_ID.– Free-form vs. structured fields: Last name, first name.– Inconsistent data values: different phone numbers.– Missing data: how will the value for insurance be located?
14
Need to Separate Operational and Informational Systems
• Operational system used to run a business in real time based on current data.
– E.g. sales order processing, reservation systems, patient registration, – Process large volumes of relatively simple read/write transactions,
while providing fast response.
• Information systems designed to support decision making based on historical data.
– Designed for complex and read-only queries or data mining application.
– Sales trend analysis, customer segmentation, and human resource planning.
15
Need to Separate Operational and Informational Systems (2)
• It is essential to separate informational processing from operational processing by creating a data warehouse.– A DW centralizes data (at least logically) that are scattered
throughout disparate operational systems and makes them readily available for decision support.
– A properly designed DW adds value to data by improving their quality and consistency.
– A separate data warehouse eliminates much of the contention for resources that results when informational applications are cofounded with operational processing.
16
Data Warehouse vs. Operational DB Systems
• OLTP (on-line transaction processing)– Major task of traditional relational DBMS– Day-to-day operations: purchasing, inventory,
banking, manufacturing, payroll, registration, accounting, etc.
• OLAP (on-line analytical processing)– Major task of data warehouse system– Data analysis and decision making
17
Data Warehouse vs. Operational DB Systems
• Distinct features (OLTP vs. OLAP):– User and system orientation: customer vs. market– Data contents: current, detailed vs. historical,
consolidated– Database design: ER + application vs. star +
subject– View: current, local vs. evolutionary, integrated– Access patterns: update vs. read-only but complex
queries
18
Why Separate Data Warehouse?• High performance for both systems:
– DBMS — tuned for OLTP: access methods, indexing, concurrency control, recovery
– Warehouse — tuned for OLAP: complex OLAP queries, multidimensional view, consolidation.
• Different functions and different data:– missing data: Decision support requires historical data which
operational DBs do not typically maintain– data consolidation: DS requires consolidation (aggregation,
summarization) of data from heterogeneous sources– data quality: different sources typically use inconsistent data
representations, codes and formats which have to be reconciled.
19
Advantages of Warehousing Approach
• High query performance– But not necessarily most current information
• Doesn’t interfere with local processing at sources– Complex queries at warehouse– OLTP at information sources
• Information copied at warehouse– Can modify, annotate, summarize, restructure, etc.– Can store historical information– Security, no auditing
• Has caught on in industry
The Terms &The Definitions
21
The Data Warehouse
• Strategic response to customer requirement for providing and processing information:– at various levels of abstraction– using history for trend analysis– with high performance
• What it provides - A protected business decision support environment - A repository of consolidated corporate data - A staging area for revitalizing operational systems
22
Multi-Dimensional Database
What is A Data Warehouse
O L A P
Multi-relational tools
M e t a D a t aM e t a D a t a
D S SD S S E I SE I S
Data ScrubbersData Scrubbers
Star SchemaDim
ensional Data M
odeling
Dimensional D
ata Modelin
g
Middleware
Data M
artData Warehouse ManagerData Warehouse Manager
Data Propagation
Data Mining
Data Mining
Data Rotation
ESS
23
What is a Data Warehouse?A Practitioners Viewpoint
“A data warehouse is simply a single, complete, and consistent store of data obtained from a variety of sources and made available to end users in a way they can understand and use it in a business context.”
-- Barry Devlin, IBM Consultant
24
What is a Data Warehouse?An Alternative Viewpoint
“A DW is a – subject-oriented,– integrated,– time-varying,– non-volatile
collection of data that is used primarily in organizational decision making.”
-- W.H. Inmon, Building the Data Warehouse, 1992
25
The Data Warehouse
• Key characteristics– Subject-oriented– Integrated– Time-variant– Nonvolatile
26
Subject Oriented
Operational Applications/Databases
Data Warehouse Subjects
Order BillingAccounts ReceivableAccounts PayableLoansSavingsLife Insurance Claims ProcessingAuto Insurance
CustomerClaimsSalesProduct
• Data is stored by business subject rather than by application
27
Integrated
Data WarehouseDatabase
Subject = Customer
SavingsDatabase
SavingsApplication
Current Accounts
Application
Current AccountsDatabase
PersonalLoans
Application
Personal LoansDatabase
Operational Environment Decision Support Environment
No
Application
Flavor
• Data is stored once in a single integrated location
Customer data stored in several Databases
28
Time-variant
- Data is tagged with some element of time - creation date, as of date/to , etc.
- Data is available for long periods of time. For example, five or more years
Data Warehouse Data
Time Data {
Key, Version and Date timestamp
• Data is stored as a series of snapshots or views which records data content and context across time.
29
Non-volatile
Data WarehouseBusiness Users & Applications
Internal Source Systems
CreateUpdateDeleteTransactions
READ-ONLY
• Existing data in the warehouse is not overwritten or updated.
DataWarehouse
External Source Systems
30
How the Data Warehouse evolved
OperationalReporting
Data Extraction/Replication
Data WarehousesData Marts
OLAP ServersData Mining
31
Line of Business Data Marts extend the concept
• Extends the concept of Data Warehousing into the various lines-of-business in support of specific needs for business intelligence
Data WarehouseOther
Business Source Systems
ExternalData
Line of BusinessSystems
Data Staging/Replication Layer
Data Marts
Extraction
Transformation
Cleansing
32
Data Mining further extends the concepts of tactical access to data in support of
specific business objectives
• Specialized applications which run on OLAP servers for drill-down processing
• Can include access by neural nets, gophers and agents
Data WarehouseOther
Business Source Systems
ExternalData
Line of BusinessSystems
Data Staging/Replication Layer
Extraction
Transformation
Cleansing
33
Data Warehousing
• Definition 1:– The process of constructing and using data
warehouses
• Definition 2:– The process whereby organizations extract
meaning from their informational assets through the use of data warehouses.