Building the Operational Data Store on DB2 UDB
-
Upload
colin-alan-macguire -
Category
Documents
-
view
218 -
download
0
Transcript of Building the Operational Data Store on DB2 UDB
-
8/8/2019 Building the Operational Data Store on DB2 UDB
1/360
ibm.com/redbooks
Building the OperationalData Store on DB2 UDBUsing IBM Data Replication, WebSphere
MQ Family, and DB2 Warehouse Manager
Corinne Baragoin
Marty Marini
Cellan Morgan
Olaf Mueller
Andrew Perkins
Kiho "Henry" Yim
Addressing business issues through
ODS architectures
Solving ODS issues using IBM
Data Replication, MQSI, DB2 UDB
Populating the ODS using
DB2 Warehouse Manager
Front cover
http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/ -
8/8/2019 Building the Operational Data Store on DB2 UDB
2/360
-
8/8/2019 Building the Operational Data Store on DB2 UDB
3/360
Building the Operational Data Store on DB2 UDBUsing IBM Data Replication, WebSphere MQ Family,and DB2 Warehouse Manager
December 2001
International Technical Support Organization
SG24-6513-00
-
8/8/2019 Building the Operational Data Store on DB2 UDB
4/360
Copyright International Business Machines Corporation 2001. All rights reserved.
Note to U.S Government Users Documentation related to restricted rights Use, duplication or disclosure is subject to
restrictions set forth in GSA ADP Schedule Contract with IBM Corp.
First Edition (December 2001)
This edition applies to DB2 UDB for z/OS and OS/390 V7, DB2 UDB Extended Enterprise Editionfor Solaris and AIX V7.2, DB2 Warehouse Manager V7.2, WebSphere MQ V5.2.1, WebSphereMQ Integrator V2.0.2, DB2 DataPropagator for z/OS V5, DataJoiner V2.1.1, and IMSDataPropagator for z/OS V3.
Comments may be addressed to:IBM Corporation, International Technical Support OrganizationDept. QXXE Building 80-E2
650 Harry RoadSan Jose, California 95120-6099
When you send information to IBM, you grant IBM a non-exclusive right to use or distribute theinformation in any way it believes appropriate without incurring any obligation to you.
Take Note! Before using this information and the product it supports, be sure to read the
general information in Special notices on page xv.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
5/360
Copyright IBM Corp. 2001 iii
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Special notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
IBM trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvi
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
The team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviiiNotice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Comments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx
Chapter 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Introducing the Operational Data Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Business questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 What is an Operational Data Store (ODS)? . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1 Definition of an ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.2 Characteristics of an ODS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.3 Benefits of an ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.4 When to use a separate datastore? . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3.5 Some differences between an ODS and a DW . . . . . . . . . . . . . . . . . 8
1.4 How to position the ODS within the BI architecture. . . . . . . . . . . . . . . . . . 13
Chapter 2. ODS issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.1 The business requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Mapping the requirements to the ODS issues. . . . . . . . . . . . . . . . . . . . . . 19
2.3 Transferring the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Integrating multiple operational sources . . . . . . . . . . . . . . . . . . . . . . 23
2.3.2 ODS data transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4 Data characteristics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.1 Near-current data delivery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.2 Source synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.3 Current data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.4 Level of detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5 The ODS environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.5.1 Mixing updates and queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 Flexible growth path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.5.3 Uptime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
-
8/8/2019 Building the Operational Data Store on DB2 UDB
6/360
iv Building the Operational Data Store on DB2 UDB
2.6 ODS administration and maintenance. . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6.1 Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.6.2 Performance tuning and monitoring . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.6.3 Initial loads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Chapter 3. ODS architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1 Business scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.1.1 Consolidated customer and product information. . . . . . . . . . . . . . . . 34
3.1.2 Consolidated customer and order information . . . . . . . . . . . . . . . . . 35
3.1.3 Detect calling card fraud while the call is in progress . . . . . . . . . . . . 37
3.2 ODS types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.2.1 ODS type A: real-time data access/localized update . . . . . . . . . . . . 43
3.2.2 ODS type B: update transaction trigger . . . . . . . . . . . . . . . . . . . . . . 46
3.2.3 ODS type C: fully integrated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493.2.4 Selecting the right type of ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3 Data modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.3.1 ERM modeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.3.2 Dimensional modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3.3 Deciding on the modeling technique . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4 ODS layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.4.1 Defining the operational data needed . . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.2 Data acquisition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.4.3 ODS workload considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 633.4.4 Information catalog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.4.5 System management requirements . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.4.6 Data access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.5 Data acquisition scenarios using IBM products. . . . . . . . . . . . . . . . . . . . . 73
Chapter 4. Populating the ODS from relational sources . . . . . . . . . . . . . . 91
4.1 Relational data as a source (ODS type A) . . . . . . . . . . . . . . . . . . . . . . . . 92
4.1.1 The relational replication solution . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.1.2 ODS data transformations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.1.3 Near-current data delivery and uptime . . . . . . . . . . . . . . . . . . . . . . 100
4.1.4 Populating the ODS using Data Warehouse Center (DWC) . . . . . . 108
4.2 Relational data as a target (ODS type B) . . . . . . . . . . . . . . . . . . . . . . . . 121
4.2.1 Managing updates back to data sources . . . . . . . . . . . . . . . . . . . . 122
4.2.2 Independent updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Chapter 5. Populating the ODS from non-relational sources . . . . . . . . . 127
5.1 VSAM as an ODS source (ODS type A) . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.1.1 The VSAM issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.1.2 Capturing VSAM updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
5.1.3 Populating the ODS using MQSI. . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.1.4 Populating the ODS using the DWC MQ Connector . . . . . . . . . . . . 153
-
8/8/2019 Building the Operational Data Store on DB2 UDB
7/360
-
8/8/2019 Building the Operational Data Store on DB2 UDB
8/360
vi Building the Operational Data Store on DB2 UDB
8.2.4 Uptime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.2.5 Performance tuning and monitoring . . . . . . . . . . . . . . . . . . . . . . . . 252
8.2.6 Initial load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
8.3 DB2 UDB for z/OS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
8.3.1 Level of detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
8.3.2 Near-current data delivery and mixing updates and queries. . . . . . 258
8.3.3 Flexible growth path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
8.3.4 Uptime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
8.3.5 Performance tuning and monitoring . . . . . . . . . . . . . . . . . . . . . . . . 270
8.3.6 Initial load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Appendix A. Implementing message flows with MQSI . . . . . . . . . . . . . . 279
Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
Using the MQSI Control Center . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280Steps required to implement an MQSI flow . . . . . . . . . . . . . . . . . . . . . . . . . . 281
Configuring the broker domain and topology . . . . . . . . . . . . . . . . . . . . . . 281
Assigning message flows and sets to the broker . . . . . . . . . . . . . . . . . . . 286
Appendix B. Example CICS program. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
Appendix C. Population subsystem: tables; SQL stored procedures . . 293
Example tables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
Part source table in DB2 for z/OS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294PART source table in Oracle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
PART source in VSAM file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
DB2 for z/OS PART staging table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
Oracle PART staging table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
Common changed data table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
ODS PART table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Stored Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
Stored Procedure to process the change records from DB2 for z/OS . . . 297
Stored Procedure to process the change records from Oracle. . . . . . . . . 302Stored Procedure to process the change records from VSAM . . . . . . . . . 307
Stored Procedure to apply the final change records to the ODS . . . . . . . 311
Appendix D. DB2 UDB for z/OS Cross Loader . . . . . . . . . . . . . . . . . . . . . 317
Appendix E. The testing environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323Other resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324
Referenced Web sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
-
8/8/2019 Building the Operational Data Store on DB2 UDB
9/360
Contents vii
IBM Redbooks collections. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
-
8/8/2019 Building the Operational Data Store on DB2 UDB
10/360
viii Building the Operational Data Store on DB2 UDB
-
8/8/2019 Building the Operational Data Store on DB2 UDB
11/360
Copyright IBM Corp. 2001 ix
Figures
1-1 Positioning the ODS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1-2 Updating/loading of data into ODS and DW . . . . . . . . . . . . . . . . . . . . . 10
1-3 IBM business intelligence architecture. . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-1 ODS architecture overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3-1 Consolidated customer portfolio data flow . . . . . . . . . . . . . . . . . . . . . . . 35
3-2 Consolidated order maintenance data flow . . . . . . . . . . . . . . . . . . . . . . 36
3-3 Consolidated call information data flow . . . . . . . . . . . . . . . . . . . . . . . . . 38
3-4 Data flow overview for ODS types A, B, and C . . . . . . . . . . . . . . . . . . . 39
3-5 Architecture of ODS types A, B, and C . . . . . . . . . . . . . . . . . . . . . . . . . 423-6 Architecture of an ODS type A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3-7 Architecture of an ODS type B. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3-8 ODS Type B trigger update process . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3-9 Architecture of an ODS type C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3-10 Selecting the right type of ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3-11 ERM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3-12 Star schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3-13 ODS layers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3-14 ODS data source from the operational data . . . . . . . . . . . . . . . . . . . . . 593-15 Data acquisition components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3-16 Metadata architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3-17 Example data access scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3-18 DB2 DataPropagator Relational and DataJoiner . . . . . . . . . . . . . . . . . . 74
3-19 IMS DataPropagator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3-20 DataRefresher and DDU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3-21 MQPUT and MQGET. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
3-22 Writing to a remote queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783-23 DB2 Warehouse Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3-24 IBM product scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3-25 IBM products for an ODS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3-26 ODS type A with relational data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3-27 ODS type B on DB2 UDB for z/OS target with relational data . . . . . . . . 86
3-28 ODS type B on DB2 UDB EEE target with relational data . . . . . . . . . . . 87
3-29 ODS type A on DB2 UDB for z/OS target with non-relational data . . . . 88
3-30 ODS type A on DB2 UDB EEE target with non-relational data . . . . . . . 89
3-31 ODS type B on with non-relational data. . . . . . . . . . . . . . . . . . . . . . . . . 904-1 UOW table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4-2 Spill file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4-3 Homogeneous replication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
-
8/8/2019 Building the Operational Data Store on DB2 UDB
12/360
x Building the Operational Data Store on DB2 UDB
4-4 Heterogeneous replication with DataJoiner . . . . . . . . . . . . . . . . . . . . . . 96
4-5 Apply pull/push modes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4-6 Apply tasks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4-7 CCD table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4-8 Overview of PART replication process . . . . . . . . . . . . . . . . . . . . . . . . 109
4-9 Source databases are registered in the DWC . . . . . . . . . . . . . . . . . . . 112
4-10 Define the DB2 replication source and select only replication sources 113
4-11 Define the DB2 replication source: import the source metadata . . . . . 114
4-12 Selecting sources to be added to the process modeler canvas. . . . . . 115
4-13 Defining a replication step: selecting staging table parameters. . . . . . 116
4-14 Defining a replication step: generating the output table. . . . . . . . . . . . 117
4-15 Defining a replication step: mapping columns to generated table . . . . 118
4-16 Defining a replication step: processing options . . . . . . . . . . . . . . . . . . 119
4-17 Defining a replication step: the final workflow . . . . . . . . . . . . . . . . . . . 1204-18 ODS type B: updating back to relational database . . . . . . . . . . . . . . . 123
4-19 MQSI message flows for type B (relational). . . . . . . . . . . . . . . . . . . . . 124
5-1 Near-real-time updates to the ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5-2 Synchronous update . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
5-3 Asynchronous update on DB2 UDB for z/OS . . . . . . . . . . . . . . . . . . . 131
5-4 Using MQSI to read the MQ queue . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
5-5 Using MQSI on distributed platform . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
5-6 DB2MQ function. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5-7 Creating an MQSI message flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5-8 Schematic message flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5-9 Deploying the flow to remote machines . . . . . . . . . . . . . . . . . . . . . . . . 138
5-10 MQInput node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5-11 MQOutput node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5-12 The compute node . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
5-13 Schematic diagram of message flow . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5-14 Updating the ODS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5-15 Multi-broker domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1475-16 Simple compute node throughput for Solaris . . . . . . . . . . . . . . . . . . . . 149
5-17 Database node throughput for Solaris . . . . . . . . . . . . . . . . . . . . . . . . . 150
5-18 Simple Compute node throughput for AIX . . . . . . . . . . . . . . . . . . . . . . 152
5-19 Database node throughput for AIX . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5-20 Population process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
5-21 Using the DWC to create an MQSeries source . . . . . . . . . . . . . . . . . . 157
5-22 MQ-Assist Wizard, selecting the type of read access . . . . . . . . . . . . . 158
5-23 MQ-Assist Wizard, naming the UDF . . . . . . . . . . . . . . . . . . . . . . . . . . 158
5-24 MQ-Assist Wizard, providing the service point name . . . . . . . . . . . . . 1595-25 MQ-Assist Wizard, specifying the message format type . . . . . . . . . . . 159
5-26 MQ-Assist Wizard, column definition page . . . . . . . . . . . . . . . . . . . . . 160
5-27 Using DWC SQL step to stage MQ messages into a relational table . 161
-
8/8/2019 Building the Operational Data Store on DB2 UDB
13/360
Figures xi
5-28 Workflow to bring orders into the ODS . . . . . . . . . . . . . . . . . . . . . . . . 162
5-29 Synchronous IMS-to-DB2 Propagation . . . . . . . . . . . . . . . . . . . . . . . . 167
5-30 Asynchronous IMS log to DB2 replication . . . . . . . . . . . . . . . . . . . . . . 168
5-31 Concept of MQ-ASYNC replication . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
5-32 Near-real-time replication using MQSeries . . . . . . . . . . . . . . . . . . . . . 171
5-33 DB2 UDB EEE as the target ODS from IMS DPROP . . . . . . . . . . . . . 172
5-34 IMS DPROP mapping concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
5-35 ODS type B flow for non-relational data. . . . . . . . . . . . . . . . . . . . . . . . 175
5-36 MQSI flow to operational database . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
6-1 Reject data analysis for a business unit at ABC Bank. . . . . . . . . . . . . 181
6-2 Rejected record not fixed. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
6-3 Rejected record fixed before system of record update . . . . . . . . . . . . 182
6-4 Components for an ODS (type A) reject handling application . . . . . . . 184
7-1 The population subsystem reference architecture . . . . . . . . . . . . . . . . 1927-2 High-level view of our PARTs population process. . . . . . . . . . . . . . . . 193
7-3 The Extract component of our PARTs ODS . . . . . . . . . . . . . . . . . . . . 195
7-4 Adding the prepare component to our PARTs ODS . . . . . . . . . . . . . . 196
7-5 Adding the transform component to our PARTs ODS . . . . . . . . . . . . . 198
7-6 Adding the load component to our PARTs ODS . . . . . . . . . . . . . . . . . 200
7-7 Independently executing units of work. . . . . . . . . . . . . . . . . . . . . . . . . 201
7-8 The DB2 replication workflow as seen in the DWC . . . . . . . . . . . . . . . 204
7-9 DWC replication step properties: replication page. . . . . . . . . . . . . . . . 205
7-10 DWC replication step properties: parameters page . . . . . . . . . . . . . . . 206
7-11 DWC replication step properties: column mapping page. . . . . . . . . . . 207
7-12 DWC replication step properties: processing options page . . . . . . . . . 208
7-13 IBM DB2 Stored Procedure Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
7-14 The Oracle replication workflow as seen in the DWC . . . . . . . . . . . . . 210
7-15 Schedule of the first step in the Oracle replication workflow . . . . . . . . 212
7-16 The VSAM PARTs via MQ workflow as seen in the DWC. . . . . . . . . . 213
7-17 Schedule of the VSAM MQ Series workflow . . . . . . . . . . . . . . . . . . . . 214
7-18 The workflow to update the ODS as seen in the DWC . . . . . . . . . . . . 2147-19 Schedule of the load ODS workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 215
7-20 The entire ODS population subsystem . . . . . . . . . . . . . . . . . . . . . . . . 216
7-21 Metadata flows using the DWC and the Information Catalog Manager 218
7-22 Metadata execution using the DWC work-in-progress window . . . . . . 220
7-23 Many objects can be defined in the information catalog . . . . . . . . . . . 221
7-24 Our ODS subjects in the information catalog. . . . . . . . . . . . . . . . . . . . 222
7-25 Tracing our ODS PART table back to its sources . . . . . . . . . . . . . . . . 223
7-26 Starting the publish DWC metadata dialog . . . . . . . . . . . . . . . . . . . . . 224
7-27 Selecting the processes, sources, and targets to publish . . . . . . . . . . 2257-28 Publish metadata processing log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
7-29 Scheduling future metadata updates to be published automatically . . 227
8-1 ODS synchronization points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
-
8/8/2019 Building the Operational Data Store on DB2 UDB
14/360
xii Building the Operational Data Store on DB2 UDB
8-2 Single partition database compared to a multi-partition database . . . . 235
8-3 Sample MPP and SMP configurations. . . . . . . . . . . . . . . . . . . . . . . . . 236
8-4 DB2 UDB EEE table partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
8-5 Three types of collocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
8-6 Intra-partition parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8-7 Inter-partition parallelism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
8-8 Combining inter-partition and intra-partition parallelism. . . . . . . . . . . . 244
8-9 SMP scalability with DB2 UDB EE. . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
8-10 SMP and MPP scalability with DB2 UDB EEE. . . . . . . . . . . . . . . . . . . 247
8-11 DB2 UDB EEE on a large SMP box. . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8-12 Partitioned table granularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
8-13 Parallel Sysplex cluster architecture and ODS . . . . . . . . . . . . . . . . . . 266
8-14 Parallel Sysplex with data sharing improves availability . . . . . . . . . . . 268
A-1 Control center panel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280A-2 Creating a broker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
A-3 PART input file. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
A-4 Defining a message set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
A-5 Defining a message flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
A-6 Designing the message flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
A-7 Deploying the message flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
C-1 Table definition for the PART table in the DB2 for z/OS database . . . 294
C-2 Table definition for the PART table in the Oracle database. . . . . . . . . 294
C-3 Relational view of the MQ queue representing the VSAM PART file. . 295C-4 Table used for staging change records from the DB2 PART table . . . 295
C-5 Table used for staging change records from the Oracle PART table . 296
C-6 Table used to store the common change records . . . . . . . . . . . . . . . . 296
C-7 The final consolidated PART table in the ODS . . . . . . . . . . . . . . . . . . 297
D-1 Cross Loader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
8-15 Testing environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
8-16 Parts-r-Us entity-relationship diagram . . . . . . . . . . . . . . . . . . . . . . . . . 322
-
8/8/2019 Building the Operational Data Store on DB2 UDB
15/360
Copyright IBM Corp. 2001 xiii
Tables
1-1 Differences between ODS and DW . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2-1 Business questions and requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2-2 Relationships between business requirements and ODS issues. . . . . . 20
3-1 Characteristics for the general ODS types A, B and C . . . . . . . . . . . . . 40
3-2 Sample users groups and their requirements . . . . . . . . . . . . . . . . . . . . 72
4-1 Decrease of connections using subscription sets (part 1) . . . . . . . . . . 106
4-2 Minimizing database connections using subscription sets (part 2) . . . 106
D-1 Cross-Loader statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
-
8/8/2019 Building the Operational Data Store on DB2 UDB
16/360
xiv Building the Operational Data Store on DB2 UDB
-
8/8/2019 Building the Operational Data Store on DB2 UDB
17/360
Copyright IBM Corp. 2001 xv
Special notices
References in this publication to IBM products, programs or services do not implythat IBM intends to make these available in all countries in which IBM operates.
Any reference to an IBM product, program, or service is not intended to state or
imply that only IBM's product, program, or service may be used. Any functionally
equivalent program that does not infringe any of IBM's intellectual property rights
may be used instead of the IBM product, program or service.
Information in this book was developed in conjunction with use of the equipment
specified, and is limited in application to those specific hardware and software
products and levels.
IBM may have patents or pending patent applications covering subject matter in
this document. The furnishing of this document does not give you any license to
these patents. You can send license inquiries, in writing, to the IBM Director of
Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of theinformation which has been exchanged, should contact IBM Corporation, Dept.
600A, Mail Drop 1329, Somers, NY 10589 USA.
Such information may be available, subject to appropriate terms and conditions,
including in some cases, payment of a fee.
The information contained in this document has not been submitted to any formal
IBM test and is distributed AS IS. The use of this information or the
implementation of any of these techniques is a customer responsibility anddepends on the customer's ability to evaluate and integrate them into the
customer's operational environment. While each item may have been reviewed
by IBM for accuracy in a specific situation, there is no guarantee that the same or
similar results will be obtained elsewhere. Customers attempting to adapt these
techniques to their own environments do so at their own risk.
Any pointers in this publication to external Web sites are provided for
convenience only and do not in any manner serve as an endorsement of these
Web sites.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
18/360
xvi Building the Operational Data Store on DB2 UDB
IBM trademarksThe following terms are trademarks of the International Business Machines
Corporation in the United States and/or other countries:
e (logo)
Redbooks (logo)
AIX
AIX 5L
AS/400
Balance
CICS
Cross-Site
CS SystemsCurrent
DataJoiner
DataPropagator
DataRefresher
DB2
DB2 OLAP Server
DB2 Universal Database
DRDA
Enterprise Storage Server
IBM
IMS
Information Warehouse
Informix
MQSeries
MVS
MVS/ESANetfinity
NetView
Notes
OS/390
OS/400
Parallel Sysplex
Perform
Planet Tivoli
QMF
Redbooks
RS/6000
S/390
Sequent
SP
SupportPac
TivoliTivoli Enterprise
TME
Visual Warehouse
WebSphere
Working Together
z/OS
Other company trademarks
The following terms are trademarks of other companies:
C-bus is a trademark of Corollary, Inc. in the United States and/or other countries.
Java and all Java-based trademarks and logos are trademarks or registered trademarks
of Sun Microsystems, Inc. in the United States and/or other countries.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft
Corporation in the United States and/or other countries.
PC Direct is a trademark of Ziff Communications Company in the United States and/or
other countries and is used by IBM Corporation under license.
ActionMedia, LANDesk, MMX, Pentium and ProShare are trademarks of Intel
Corporation in the United States and/or other countries.
UNIX is a registered trademark in the United States and other countries licensed
exclusively through The Open Group.
SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned bySET Secure Electronic Transaction LLC.
Other company, product, and service names may be trademarks or service marks of
others.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
19/360
Copyright IBM Corp. 2001 xvii
Preface
For the past several years many companies have undergone reorganizationsand process re-engineering activities to break down their vertical business
processes. A more customer-centric and cross functional approach is being
taken to increase customer service and reduce costs. The challenge for
Information Technology (IT) departments is to integrate and consolidate the
required data across many different systems.
This IBM Redbook focuses on how an Operational Data Store (ODS) can be
used to satisfy these requirements. We define what an ODS is and the types of
business questions it can answer. The ODS is positioned within the BusinessIntelligence (BI) architecture and specific issues are discussed. Three distinctive
business scenarios are used to help explain the ODS architecture.
Using business examples, this book describes the different methods, techniques,
and technologies used to implement an ODS:
IBM Data Replication with DB2 DataPropagator (DB2 DPROP throughout
the book), DataJoiner, and IMS DataPropagator (IMS DPROP throughout the
book)
WebSphere MQ (MQSeries throughout the book) and WebSphere MQ
Integrator (MQSI throughout the book)
DB2 Warehouse Manager (DB2 WM throughout the book) and Data
Warehouse Center (DWC throughout the book)
These IBM products are used to describe how the ODS is populated. A number
of source and target scenarios are used to demonstrate both homogeneous and
heterogeneous environments. DB2 Warehouse Manager is also used to manage
and control the population of the ODS and both DB2 UDB EEE for UNIX andDB2 UDB for z/OS and OS/390 (respectively termed DB2 EEE and DB2 for z/OS
throughout the book) are presented as targets for the ODS.
This book can be used by business decision makers, architects, implementors,
DBAs, and data acquisition specialists to gain a better understanding of what an
ODS is, how it can be used, and how it should be designed and implemented.
This book makes the assumption that the reader is familiar with many of the
issues and characteristics of a data warehouse and its architecture. Using datawarehousing terms and techniques, we describe the differences and similarities
between the two enterprise data stores: the data warehouse and the ODS.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
20/360
xviii Building the Operational Data Store on DB2 UDB
The team that wrote this redbook
This redbook was produced by a team of specialists from around the world
working at the International Technical Support Organization, San Jose Center.
Corinne Baragoin is a Business Intelligence Project Leader with the IBMInternational Technical Support Organization, San Jose Center. She has over 16
years of experience as an IT specialist on DB2 and related solutions. Before
joining the ITSO, she had been working as an IT Specialist for IBM France,
assisting customers on DB2 UDB and data warehouse environments.
Marty Marini is a Senior IT Specialist with the Business Intelligence Practice in
Toronto, Canada. His main areas of expertise include DB2 UDB EEE, AIX, and
RS/6000 SP systems. He is a Data Management Specialist and is a Certified
Advanced Technical Expert for DB2 Clusters and a Certified Solutions Expert forDB2 Administration and DB2 Application Development. Marty holds a degree in
Computer Science with 14 years of experience in Database and Application
Development projects.
Cellan Morgan is a Systems Specialist in the United Kingdom. He has 14 years
of experience as a CICS Systems Programmer and two years experience as an
MQSeries Systems Programmer and Support Specialist. Cellan provides support
for both IBM internal systems and external customers.
Olaf Mueller is an IT Specialist with IBM Global Services, e-Business Integration
in Duesseldorf, Germany. He has 13 years of experience as a Database
Specialist working in the areas of application development and database
administration. He is a Certified Solutions Expert for DB2 Administration and
DB2 Application Development. His areas of expertise include Oracle and data
modeling.
Andrew Perkins is a Certified Consulting IT Specialist supporting Business
Intelligence technical presales activities in the IBM Americas BusinessIntelligence Solutions Center in Dallas, TX. He has over 20 years of experience
in the IT field, concentrating primarily on DB2 and related solutions. He holds a
Bachelor of Science degree in Computer Science from the University of Houston.
His areas of expertise include DB2 UDB across all platforms, DB2 Warehouse
Manager, and DB2 OLAP Server.
Kiho "Henry" Yim is a production DBA at Sterling Forest, NY, IBM Global
Services. He joined IBM Korea in 1976 as a Systems Engineer and moved to the
United States in 1980. He has held various positions, including DBA and
Database Developer for DB2, IDMS, ADABAS, IMS and systems programmer for
CICS. He is an IBM certified IT specialist and holds an EE degree from Seoul
National University, Korea.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
21/360
Preface xix
We would like to specially thank Mark Persaud for his contributions in writing
some sections of the book:
Mark Persaud is an IT Architect with the Data Warehouse Practice in Toronto,
Canada. Mark has 10 years of design and development experience which span
data warehouse, ODS, and traditional OLTP system implementations. He workedfor more than 6 years in one of Canada's largest banks as a designer and
developer of OLTP banking systems. Currently, Mark is focused on working with
our clients in the field to define solution architectures for data warehouse and
ODS related initiatives.
We would like also to thank Chris Todd and IBM Global Services (IGS) for
providing much of the ODS architecture material. The IGS ODS offering was
used as a foundation for this redbook.
Thanks to the following people for their contributions to this project:
By providing their architectural input and/or reviewing this redbook:
Greg Carroll
Ted Wong
Chris Todd
IBM Canada, Data Warehousing Practice
John McPherson
IBM Silicon Valley Lab
By providing their technical input and/or reviewing this redbook:
Glen Sheffield
IBM Toronto Lab
Julie Jerves
IBM Silicon Valley Lab
Rick Long
IBM AustraliaDavid Bryant
Henry Cook
Simon Harris
Simon Woodcock
IBM UK
John Bell
IBM Kansas City
Tom BauchIBM Dallas
Inderpal Narang
IBM Almaden Research Center
-
8/8/2019 Building the Operational Data Store on DB2 UDB
22/360
xx Building the Operational Data Store on DB2 UDB
Geert Van De Putte
Phil Wakelin
Bob Haimowitz
International Technical Support Organization
Notice
This publication is intended to help business users, architects, DBAs, and data
acquisition specialists understand the sorts of business issues addressed by an
ODS and to help implementors understand the ODS issues. The information in
this publication is not intended as the specification of any programming
interfaces that are provided by:
WebSphere MQ and WebSphere MQ Integrator DB2 DataPropagator for z/OS
IMS DataPropagator for z/OS
DataJoiner
DB2 Warehouse Manager
IBM DB2 Universal Database
See the PUBLICATIONS section of the IBM Programming Announcement for the
product name(s) above for more information about what publications are
considered to be product documentation.
Comments welcome
Your comments are important to us!
We want our IBM Redbooks to be as helpful as possible. Send us your
comments about this or other Redbooks in one of the following ways:
Use the online Contact us review redbook form found at:
ibm.com/redbooks
Send your comments in an Internet note to:
Mail your comments to the address on page ii.
http://www.redbooks.ibm.com/http://www.ibm.com/redbooks/http://www.redbooks.ibm.com/contacts.htmlhttp://www.redbooks.ibm.com/contacts.htmlhttp://www.ibm.com/redbooks/http://www.ibm.com/redbooks/http://www.redbooks.ibm.com/ -
8/8/2019 Building the Operational Data Store on DB2 UDB
23/360
Copyright IBM Corp. 2001 1
Chapter 1. Introduction
Throughout history, successful organizations have managed to consistently
evolve with the changing conditions of the business environment. The
environment is changing at an exponential rate, causing organizations tocontinually invent new and innovative ideas at an even more rapid pace. It is a
necessity that organizations adapt, evolve, and innovate to stay competitive.
Over the past few years, organizations have routinely used Business Intelligence
and Data Warehousing techniques to gather data together into a single
integrated view, to store historical data to and to make strategic business
decisions by analyzing this data. This used to be a background process.
However, organizations have become more confident in using this type of
information. They have seen increasing need to integrate data across business
processes, channels, and organizational departments. In the new economy,
customers are demanding immediate access to a fully integrated portfolio of
products and services. Organizations must quickly satisfy these demands by
changing to a more customer-centric infrastructure. In addition, they can see the
benefit of doing this in real-time.
In this chapter we introduce the Operational Data Store (ODS) and the business
challenges facing organizations in the new marketplace. We then define what an
ODS is and compare it to the DW. Finally we position the ODS within theBusiness Intelligence (BI) architecture.
1
-
8/8/2019 Building the Operational Data Store on DB2 UDB
24/360
2 Building the Operational Data Store on DB2 UDB
1.1 Introducing the Operational Data Store
It is addressing business needs that has driven the development of the
Operational Data Store (ODS), a data structure designed around a specific set of
data that is, integrated for a specific purpose. The ODS typically contains
current data which is dynamically updated in a real-time (or near-real-time)
manner. In short, an ODS is an architectural construct containing subject
oriented, integrated, volatile, current (or near-current), and detailed operational
information.
One of the questions organizations face in deploying an ODS is whether to use
their existing DW system to store the real-time updates for the ODS. While this
may work in some situations, most clients find that conflicting Service Level
Agreements (SLAs) make it more desirable to have the ODS and data
warehouse (DW) operate on distinct schemas, or even totally separate
environments. Examples of conflicting SLAs include the need to have data in the
ODS faster than it can be transformed into the standardized DW format, as well
as unresolvable business priority issues between the ODS user and the
demands placed on the DW by ad-hoc queries.
Note 1: In May of 2000, IBM Global Services announced the Operational
Data Store for real-time data access. For more information on this offering
visit the business intelligence Internet site at http://www.ibm.com/BI andhttp://www-4.ibm.com/software/data/bi/consulting/datastore.htm).
Note 2: The ODS is also a component of the Zero Latency Enterprise (ZLE),
a term coined by the Gartner Group for near-real-time integration of
operational and information systems for data and business processes. The
ZLE solution decreases the time it takes to make business decisions. In effect,
there is zero latency between the cause and effect of a business decision. A
ZLE with end-to-end quality of service involves heterogeneous andhomogeneous replication, one or more near-real-time ODSs, a data
warehouse, datamarts, and Business Intelligence applications. The ZLE
solution enables analysis across the ODS, the data warehouse, and other
sources (for example, weblogs). The ZLE also notifies the business of
actionable recommendations (using publication and subscription services),
effectively closing the loop on business processes with the operational
systems.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
25/360
Chapter 1. Introduction 3
1.2 Business questions
The emergence of the Internet and e-commerce have created some unique and
interesting opportunities and challenges. We now have the technology to transfer
information to anyone anywhere in the world. With this new capability, customers
are asking for more and more customized products and services. This hascreated competitive pressures to get these products and services to the market
in shorter and shorter time frames.
Many businesses have undergone a restructuring and reorganization to break
down their horizontal functional boundaries. Now the focus is on improving
business management and processes across traditional business functions
(for example, customer service, campaign management, risk management,
inventory, and supply chain management). Unfortunately, users must constantly
access multiple applications when dealing with these new processes.
Deploying an ODS is an excellent approach, especially when this challenge
cannot be met by the existing operational applications, the DW, or the datamarts.
The ODS is not just the data store but the population around it. The front ends
and the ODS will provide access to an integrated, consolidated picture of the
organizations information and may leverage existing IT investments. This allows
organizations to improve or close the loop on business processes.
The technology infrastructure should also be designed for the cooperation andinteroperation of processes, in order to automate information flow through the
enterprise, enable rapid accommodation to change, and integrate all customer
contacts with the organization.
The business requirements may appear to be different across the various
industries, but the underlying information requirements are similar integrated,
current, detailed, and immediately accessible. The following business questions
in the different industry sectors demonstrate this need:
Insurance
How can I provide an up-to-date view of insurance products owned by each
customer for our Customer Relationship Management (CRM) system?
How can I consolidate all the information required to solve customer
problems?
How can I decrease the turn-around time for quotes?
How can I reduce the time it takes to produce claim reports?
Retail
How can we give suppliers the ability to co-manage our inventory?
-
8/8/2019 Building the Operational Data Store on DB2 UDB
26/360
4 Building the Operational Data Store on DB2 UDB
What inventory items should I be adjusting throughout the day?
How can my customers track their own orders through the Web?
What are my customers ordering across all subsidiaries?
What is the buying potential of my customer at the point of sale?
Banking/Finance What is the complete credit picture of my customer, so I can grant an
immediate increase?
How can we provide customer service have a consolidated view of all
products and transactions?
How can we detect credit card fraud while the transaction is in progress?
What is the current consolidated profitability status of a customer?
Telecommunications
Can we identify what our Web customers are looking for in real-time?
Which calling cards are being used for fraudulent calls?
What are the current results of my campaign?
How can we quickly monitor calling patterns after the merger?
Of course, there are many other possible questions, but most of them can be
summarized as follows:
How do I provide an up-to-date view of cross functional information for aparticular business process when the data is spread across several disparate
sources?
As with a DW, the ODS should be seen as a journey and not a means to an end.
Each business challenge fitting the ODS model should be categorized (by
subject area), prioritized (by the business) and then split into manageable
projects. Your ODS architecture (along with your entire BI architecture) should be
constantly re-evaluated, taking into account the current and future evolutions of
the business environment.
1.3 What is an Operational Data Store (ODS)?
In this chapter we introduce the term ODS to get a consistent understanding of
what is meant by it. The following topics are described:
ODS definition
ODS characteristics
ODS benefits
Differences between an ODS and a Data Warehouse (DW)
-
8/8/2019 Building the Operational Data Store on DB2 UDB
27/360
Chapter 1. Introduction 5
1.3.1 Definition of an ODS
An ODS is an environment where data from different operational databases is
integrated. The purpose is to provide the end user community with an integrated
view of enterprise data. It enables the user to address operational challenges
that span over more than one business function.
Its principal key differentiators are the update frequency and the direct update
paths from applications, compared to the controlled update path of data
warehouse or datamart.
1.3.2 Characteristics of an ODS
The following characteristics apply to an ODS:
Subject oriented
The ODS may be designed not only to fulfill the requirements of the major
subjects of a corporation, but also for a specific function or application (for
example, risk management, to get a holistic view of a customer).
Integrated
Legacy systems push their detailed data through a process of transformation
into the ODS. This leads to an integrated, corporate-wide understanding of
data. This transformation process is quite similar to the transformationprocess of a DW. When dealing with multiple legacy systems, there are often
data identification and conformance issues that must be addressed (for
example, customer identification, such as which codes are used to describe
the gender of a person).
Near current data delivery
The data in the ODS is continuously being updated.
Changing data permanently requires a high frequency update. Changes
made to the underlying legacy systems must reach the ODS as fast asneeded; that is, velocity is important. Some data is needed directly; other data
can come in during a daily update. Thus, update of an ODS has both high
frequency and high velocity.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
28/360
6 Building the Operational Data Store on DB2 UDB
Current data
An ODS reflects the status of its underlying source systems. The data is quite
up-to-date. In this book we follow the architectural principle that the ODS
should contain little or no history. Usually there is sufficient data to show the
current position, and the context of the current position, for example, a recent
history of transactions to assist a call center operator. Of course, if there are
overriding business requirements, this principle may be altered. If your ODS
must contain history, you will have to make sure you have a complete
understanding of what history is required and why, and you must consider all
the ramifications; for example, sizings, archiving, and performance.
Detailed
The ODS is designed to serve the operational community and therefore iskept at a detailed level. The definition of detailed depends on the business
problem that is being solved by the ODS. The granularity of data may or may
not be the same as in the source operational system. For example, for an
individual source system, the balance of every single account is important;
but for the clerk working with the ODS, just the summarized balance of all
accounts may be important.
1.3.3 Benefits of an ODSIn this section many of the essential benefits are described that can be achieved
by using an ODS:
The ODS provides improved accessibility to critical operational data.
With an ODS, organizations have a complete view of their financial metrics
and customer transactions. This is useful for better understanding of the
customer and to make well-informed business decisions.
The ODS can provide the ability to request product and service usage data ona real or near real-time basis.
Operational reports can be generated with an improved performance in
comparison to the legacy systems.
Definitions of frequency and velocity:
Frequency is how often the ODS is updated, quite possibly fromcompletely different legacy systems, using distinct population processes,
and also takes into account the volume of updates that are occurring.
Velocity is the speed with which an update must take place from thepoint in time a legacy system change occurs, to the point in time that it
must be reflected in the ODS.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
29/360
Chapter 1. Introduction 7
It is the right place to have a central version of reference data that can be
shared among different application systems. One way could be that the
applications access the data in the ODS directly. Another way is to replicate
data changes from the ODS into the databases of the legacy systems.
The ODS can help to integrate new and existing systems.
The ODS may shorten the time required to populate a DW, because a part of
the integrated data already resides in the ODS.
1.3.4 When to use a separate datastore?
It is worth discussing briefly when to use a separate data store, the ODS rather than simply extending the use of existing systems; for example, customer
master files or a data warehouse.
There is no clear-cut rule for this, and in some cases, it is possible to meet new
new real-time requirements by using existing systems.
Many tasks of a system designer or architect involve making trade-offs between
conflicting requirements, as well as anticipating how these requirements are
likely to change over time. Requirements include response time, throughput,
ease of development, cost, service levels, resource utilization, availability, fault
tolerance, serviceability, and so on.
An application database that is extended to provide ODS type function is likely to
suffer from constraints imposed because it was originally designed as an
application specific database. This may limit its performance, or if the ODS is
changed to support ODS function, this may adversely impact the original
application.
A data warehouse that is used to support more and more ODS style function islikely to become very costly, and you may find it increasingly difficult to meet
service level commitments. For example, the DW may have to be hardened to
make it, or a substantial subset of it, suitable for 24x7 operation.
It is often easier to meet these requirements with a separate dedicated data
store, the ODS, albeit at the expense of some data redundancy. The cost of the
data redundancy is more than outweighed by other savings in systems cost and
development time.
Attention: Not all ODSs will provide all these benefits.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
30/360
8 Building the Operational Data Store on DB2 UDB
It is also often easier to implement new requirements when an ODS and a DW
are used. There is usually an obvious home for the requirement: this speeds
development and deployment and gets the application to market faster, thus
delivering benefit earlier. ODS data is operational and is expected to reliably
describe what is happening down to the most detailed level. Mainstream
business processes will want to rely upon it, and the update and maintenanceprocedures should reflect this. The DW, on the other hand, may be targeted
more at strategic decision-making, where some degree of imprecision is
tolerated, the tolerance and the velocity of the data must be taken into account
before making use of it online.
In addition, the workloads for an ODS and DW are usually drastically different.
Mixing the two workloads can require more resources, and cost, for a combined
system, and can be more difficult to guarantee service levels, than if keeping
them apart. The more contention there is between workloads, the more systemresources are needed to cope with contention, in addition to coping with each
workload.
It is possible to move DWs towards real-time working, that is through trickle feed,
but often they cannot also be extended to guarantee fast and assured response
times for high volumes of transactions.
1.3.5 Some differences between an ODS and a DW
The ODS may look like a DW, but it is actually quite different. Some of the main
differences are described below.
Currency of dataAn ODS is a current view of the data. It contains current or near-current data. It
may be accurate nearly up-to-the-second, but does not have to meet this
standard. That depends on the objectives of the ODS, the user needs, and the
business requirements for velocity. For example, a risk management ODS in an
investment house may only be updated once to cover off exposures in the
overseas markets. If a business condition changes, for example, parts are taken
from stock, then the operational data related to that condition quickly changes
too. The ODS reflects the current status of a business condition, in this case, the
number of parts in stock.
The DW data is quite different. It reflects a more historical view of the business
conditions. If there are changes, a new snapshot may be created and put in line
with the other snapshots of this business condition. Insertions to the DW
(snapshots, events) are usually batched and enter the DW only according to apre-defined schedule. There is usually a variable elapsed period of time during
which the DW is not representative of the current operational situation.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
31/360
Chapter 1. Introduction 9
As Figure 1-1 shows, the ODS addresses the tactical operational data needs of
an organization by providing current or near-current integrated information,
whereas the data warehouse supports the informational or strategic needs of an
organization by providing more historical data.
Figure 1-1 Positioning the ODS
Updating/loading dataIn an ODS, altering (insert, update, delete) an existing record is a normal and
often-used process, because just one status of a business condition is reflected.
In the DW, the records are not updated, but a new snapshot or event may be
created. This difference is shown in Figure 1-2. Multiple versions of data mayexist and this depends on requirements or purpose.
Data Warehouse Operational System
Current &Historical
Current &Near Current
Consolidated,Integrated Data
Immediate accessto detailed
Transaction Data
Operational
Data
Store
-
8/8/2019 Building the Operational Data Store on DB2 UDB
32/360
10 Building the Operational Data Store on DB2 UDB
Figure 1-2 Updating/loading of data into ODS and DW
Another difference, when updating the data, is the point in time when the update
is done. An ODS needs updates as soon as possible, depending on business
requirements. This means there may be the need of a real-time or near real-time
update mechanism; this is one of the outstanding characteristics of the ODS.
Updates for a DW are normally collected, and at a predefined point in time, a
load is done into the DW.
Summarization of dataThe ODS contains primarily detailed data. It may contain summarized data for
performance reasons, but this is not its first objective, because it is difficult tokeep the summary data current and accurate.
Therefore the summarized data is normally calculated at request. Because of its
short effective life it can be called dynamic summary data. Its value depends on
the moment of calculation. For example, consider the collective account balance
of a company. This is calculated as a summary of all single accounts of the
company. If any of the single accounts changes its value, the summary also
changes.
However, certain widely-used summary totals may be maintained in the ODS.
Note: An ODS is updated based on requirements and not necessarily as soon
as possible; that might be once an hour. In fact, an ODS could be updatedovernight, but is updated directly from a front-office application such as Siebel.
ODS DW
SnapshotsInsert
Delete
Update
Delete
Insert
Update
Load
Load
Load
-
8/8/2019 Building the Operational Data Store on DB2 UDB
33/360
As an example an operational transaction might get a list of all accounts of a
-
8/8/2019 Building the Operational Data Store on DB2 UDB
34/360
12 Building the Operational Data Store on DB2 UDB
As an example, an operational transaction might get a list of all accounts of a
customer and their balances. A DW unit of work could calculate the difference
between the sum of all credits and the sum of all collateral of all customers of a
bank.
UsageThe ODS is used for day-to-day operational decisions. For example in an
insurance company the agent gets the contracts for a customer to decide which
additional products he can offer.
The insurance company uses the DW, in contrast to the ODS, to get strategic
and long-term and corporate-wide decisions. For example, the company might
decide to start a marketing campaign for certain products that had lower sales
last year.
ODS is tactical, whereas DW is strategic.
UsersBecause of the different tasks an ODS and a DW fulfill, there are also different
working communities that use these systems.
The ODS is used by the tactical community. This includes people such as
insurance agents who have daily contact with their customers and who need
information at their fingertips. The DW is more often used by the strategiccommunity. These are people who make decisions for example, a manager in
an insurance company who is responsible for the life insurance branch, and who
wants to measure the amount of insurance growth during the last five years.
Data volumeOne of the major differences is data volume: the ODS will typically contain less
data than the DW:
The ODS may contain the current value of each customer s balance(by product) to reflect the collective corporate-wide balance of a customer.
In contrast, the DW may store historical snapshots of each customer s
balance.
The ODS would contain the details of individual order line items, whereas the
DW may have only the order headers.
The ODS could store the customer contact names for the marketing,
accounting, and purchasing departments. The DW may store the history of
the customer contact names for the past five years.
Summary
-
8/8/2019 Building the Operational Data Store on DB2 UDB
35/360
Chapter 1. Introduction 13
SummaryTable 1-1 is a summary of the various differences between the ODS and a DW.
Table 1-1 Differences between ODS and DW
1.4 How to position the ODS within the BI architecture
Businesses today are faced with an extremely competitive market place. More
and more organizations are relying on Information Technology (IT) to sharpen
their competitive edge by adapting to the rapidly changing business conditions.To adapt, they are looking towards business intelligence application solutions.
There have been many successful implementations of data warehousing
solutions. The DW supports the informational and strategic needs of an
organization by providing historical snapshot data. An ODS, on the other hand,
addresses the operational data needs of an organization by providing current, or
near-current integrated information which can be directly accessed or updated by
the users.
ODS DW
Data of high quality at detailed level and
assured availability
Data may not be perfect, but sufficient for
strategic analysts; data does not have to
be highly available
Contains current and near-current data Contains historical data
Real-time and near real-time data loads Normally batch data loads
Mostly updated at data field level (even if
it may be appended)
Data is appended, not updated
Typically detailed data only Contains summarized and detailed data
Modeled to support rapid data updates
(3NF)
Variety of modeling techniques used
typically 3NF for DW and dimensional for
datamarts to optimize query performance
Transactions similar to an Online
Transaction Processing (OLTP) system
Queries process larger volumes of data
Used for detailed decision making and
operational reporting
Used for long-term decision making and
management reporting
Used at the operational level Used at the managerial level
Figure 1-3 shows how the ODS is positioned within the IBM business intelligence
-
8/8/2019 Building the Operational Data Store on DB2 UDB
36/360
14 Building the Operational Data Store on DB2 UDB
Figure 1 3 shows how the ODS is positioned within the IBM business intelligence
architecture. The ODS is an architectural structure that is populated from the
operational and external data sources through the integration and transformation
programs.
Figure 1-3 IBM business intelligence architecture
When the operational and external data passes through the integration and
transformation layer, there will be two different flows, one to the ODS and the
other to the DW. These two streams satisfy the different information needs of thevarious end users across different functions.
Data Warehouse
Integration and Transformation
Business Intelligence applications
Operational and External Data
Query&
Reports
DataMining
Statistics
OLAP
Data Marts
O D S
The ODS can push data to the DW using normal batch methods. For example,
-
8/8/2019 Building the Operational Data Store on DB2 UDB
37/360
Chapter 1. Introduction 15
p g p ,
the current value of an integrated data element could be sent to the DW for
snapshot, aging, or historical purposes.
Depending on the ODS type, a small amount of data may flow back in a
controlled fashion to the ODS from the DW. Even though this flow makes up a
very small part of the ODS, the business importance may be significant. For
example, when the DW is used to analyze and calculate a competitive interest
rate or to alter a production schedule based on the best performing items, the
results are passed back to the ODS for the purpose of making corporate-wide
operational business decisions.
Depending on the ODS type, updates to the ODS by end-users may trigger data
to flow back to the operational data sources. A typical example would be a
customers shipping address. The address will be stored in both the ODS and the
data source, however, the address in the ODS may be more up to date. In these
cases the ODS should also institute a mechanism in the integration and
transformation programs to filter these cyclical updates and stop them from
flowing back needlessly to the ODS.
In general, operational access to the ODS will occur through custom interfaces.
What we traditionally think of as query tools (QMF, Business Objects, Brio, and
so on) are not the main thrust. They might be deployed, but the idea of
operational applications must also be considered.
-
8/8/2019 Building the Operational Data Store on DB2 UDB
38/360
16 Building the Operational Data Store on DB2 UDB
-
8/8/2019 Building the Operational Data Store on DB2 UDB
39/360
Copyright IBM Corp. 2001 17
Chapter 2. ODS issues
To maximize customer satisfaction and profitability, the ultimate data store would
contain all of the organizations operational data. This, of course, is not
economical or technically feasible.
To tackle these challenges, an ODS must ultimately be constrained based on
business priorities, and the technology is being challenged to step up to these
real-time and near-real-time requirements. The existence of these priorities and
requirements drives cost and complexity. It is with these constraints in mind that
the ODS issues can be properly addressed.
At first glance, it would appear that the ODS presents a paradox. It seems to
have the characteristics of an On-Line Transactional Processing (OLTP) system
while at the same time accommodating some of the attributes of a datawarehouse (for example, integrating and transforming data from multiple
sources).
In this chapter we look at many of the issues surrounding an ODS. First, we
utilize elements from each business question,as developed in 1.2, Business
questions on page 3, to apply specific business requirements. We then map
these business requirements to their related ODS issues. In the remainder of the
chapter we categorize and explore each issue in more detail.
2
-
8/8/2019 Building the Operational Data Store on DB2 UDB
40/360
Business question Business requirements
-
8/8/2019 Building the Operational Data Store on DB2 UDB
41/360
Chapter 2. ODS issues 19
2.2 Mapping the requirements to the ODS issues
Table 2-2 defines the various relationships between the business requirements
and the ODS issues. The table also assigns one or more ODS issues to a
particular description, these descriptions are used throughout this book.
For this chapter, the issues are further categorized under one of four areas:
Transferring the data
Data characteristics
The ODS environment
ODS administration and maintenance
Insurance: How can I decrease the
turn-around time for quotes?
The integration of the legacy applications
will be phased in over the next four years.
Telecommunications: Can we identify
what our Web customers are looking for inreal-time?
Since the new Web application may be
accessed in any time zone, the availabilitymust be 24x7.
Telecommunications: Which calling cards
are being used for fraudulent calls?
We must have quality data to be
successful.
Banking/Finance: What is the current
consolidated profitability status of a
customer?
We need to know what calculations were
used to create the customer profitability
number.
Telecommunications: What are the current
results of my campaign?
The current status of the data in the ODS
must be known at all times.
Insurance: How can we reduce the time it
takes to produce claim reports?
We must leverage the infrastructure and
standards of our current environments.
Retail: What is the buying potential of my
customer at the point of sale?
Our customer services representatives
must consistently be able to access the
customer information within seconds.
Telecommunications: How can we quickly
monitor calling patterns after the merger?
We require access to one accounting
period of call information from eachsource system.
Table 2-2 Relationships between business requirements and ODS issues
-
8/8/2019 Building the Operational Data Store on DB2 UDB
42/360
20 Building the Operational Data Store on DB2 UDB
Business requirements ODS issues Description
Transferring the data
A consolidated view of all our
products by customer acrossseveral heterogeneous
operational applications.
What is the subject area? What
are the data source types,locations, and platforms? How
can we integrate data from
several sources?
Integrating
multipleoperational
sources
Consolidation of our three
inventory systems (obtained
from various mergers and
acquisitions) to allow online
inventory management by our
suppliers.
Should some or all of the ODS
become the single version of
the truth or the system of
record?
The new call center application
must have the ability to track
the number and type of
customer contacts.
How to manage user updates to
the ODS?
Inventory management
requires a consolidated and
consistent view of all product
codes and descriptions.
What kind of data
transformations are required?
ODS data
transformations
Data characteristics
Our call center requires all
account balances for a
customer within minutes of an
update.
How quickly must the ODS be
populated after an operational
transaction completes (for
example, seconds or minutes)?
near-current data
update (velocity)
Changes to the shipping
address using the new on-line
Web order maintenancesystem must also be applied to
the legacy order entry system.
How to automate the
synchronization of the ODS and
legacy systems?
Source
synchronization
Customer service requires 30
days of transaction information
to solve 99 percent of all
customer complaints.
What are the current and
near-current data
requirements?
Current data
delivery
We need access to invoice
line-item information across allof our subsidiary businesses to
solve customer problems
What is the level of detail
required?
Level of detail
Business requirements ODS issues Description
-
8/8/2019 Building the Operational Data Store on DB2 UDB
43/360
Chapter 2. ODS issues 21
The remaining sections of this chapter expand on each ODS issue.
The ODS environment
The query response time must
be five seconds or less during
periods of high transactionvolumes.
How to control concurrent
updates and queries? How to
handle intensive transactionrates?
Mixing updates
and queries
The integration of the legacy
applications will be phased in
over the next four years.
How to handle the growth of the
ODS (network and server)?
Flexible growth
path
Since the new Web application
may be accessed in any time
zone, the availability must be
7X24.
What are the availability
requirements and how do we
satisfy them?
Uptime
ODS administration and maintenance
We must have quality data for
this project to be successful.
Who owns the various data
elements in the ODS?
Administration
We need to know what
calculations were used to
create the customer profitability
number.
How do we know what data is in
the ODS and what it can be
used for?
The current status of the data in
the ODS must be known at all
times.
How do we control the
population of the ODS?
We must leverage the
infrastructure and standards of
our current environments.
How can we leverage the
current infrastructure and
enterprise standards?
Our customer services
representatives mustconsistently be able to access
the customer information within
seconds.
How can we maintain the
performance of the ODS?
Performance
tuning andmonitoring
We require access to one
accounting period of call
information from each source
system.
How do we initially load the
ODS?
Initial load
2.3 Transferring the data
-
8/8/2019 Building the Operational Data Store on DB2 UDB
44/360
22 Building the Operational Data Store on DB2 UDB
Populating data into the ODS is arguably the most complex activity of an ODS
system. Adding to this complexity is the potential for some of the ODS data to be
transferred back to the source systems.
As shown in Figure 2-1, the ODS architecture is split into three layers. The
operational and external data sources required to populate the ODS are found in
the data sources layer. The data acquisition layerextracts from the data sources,
integrates, consolidates, and transforms the data into the target format and loads
the data into the ODS. The enterprise data layerincludes the target ODS
database. The entire ODS system is built on a foundation based on the
administration and maintenance layers. Chapter 3, ODS architectures on
page 33, expands on this architecture by adding more layers and adding more
details for these.
Figure 2-1 ODS architecture overview
We now discuss the two primary tasks involved in moving the data from the data
sources to the ODS:
Integrating multiple operational sources
ODS data transformations
Maintenance
Data SourcesLayer
Data Acquisition
Layer
Enterprise DataLayer
ODSExtract Transform Load
Source
C
Source
A
Source
B
Administration
-
8/8/2019 Building the Operational Data Store on DB2 UDB
45/360
Note: We have defined the terms: system of recordand single version of the
-
8/8/2019 Building the Operational Data Store on DB2 UDB
46/360
24 Building the Operational Data Store on DB2 UDB
Careful planning is required for the user update portion of the ODS. It should be
based on new data requirements not existing in the current legacy applications.
One purpose for users writing to the ODS is to update the data warehouse, for
example, with a call resolution status code. An example of data existing in both
the legacy system and the ODS would be contact information, such as address
and telephone number.
The data may originally come from the legacy system; however, in the ODS, it is
updated by the users of the call center. The call center applications use the ODSas their authoritative source and want to keep it up to date. If these changes are
not applied back to the legacy system, then the transformation layer will require a
rule for these fields. For example, you may want to block temporary future
updates coming from the legacy system.
Allowing the ODS to become inconsistent with the legacy systems will compound
the current data integration problems. If an overriding business requirement must
update a common data element, then you should consider a mechanism to feed
these changes back to the operational source. In this case, the transformation
layer can filter any cyclical data so that it does not replicate back to the ODS.
That is an enterprise view. However, the ODS could also be built to support a
particular business area that needs to feed back changes; for example, the ODS
might be built to support a call center.
2.3.2 ODS data transformations
After the data has been sourced and extracted, we must apply anytransformations required by the business. These transformations give the final
touches to the integrated data, eliminating any last remnants of the operational
application silos.
truthas follows:
System of record: Multiple data stores may hold the same data elements,
such as customer name and status code. Each data store may update the
data elements without any synchronization between them. What version ofthe data is to be considered the correct version in two different systems
systems? The data store with the most trusted value for the data element
(as defined by the business) is called the system of record for that element.
Single version of the truth: One and only one data store contains the data
elements. All applications wishing to make changes to the data element
must use the defined data store. (Note that multiple read-onlycopies of the
data may exist in other data stores for performance reasons.)
Many of the types of transformations for a data warehouse are also included in
an ODS. The only limitation as to what transformations can be performed is the
l i h i h i kl h d b d d Th f
-
8/8/2019 Building the Operational Data Store on DB2 UDB
47/360
Chapter 2. ODS issues 25
latency constraint, that is, how quickly the data must be updated. The amount of
transformations that can be done is inversely related to the latency, that is the
more near-real-time the update requirements are, the fewer transformations can
be done.
Transformations include:
Cleansing
Conversions Some summarization
Decoding/encoding
Reformatting
Converting physical structures
The extent and type of transformations are dependent on the ODS data delivery
timing requirements. These requirements may be at odds with the transformation
requirements, and trade-offs may have to be made. In a real-time environment,
much of the data coming from the legacy systems may arrive as-is and shouldbe documented accordingly.
An Extract Transform Move Load (ETML) tool with the capability of handling high
update volumes should be considered. Some of this analysis may have already
been done for those companies that have functioning data warehouses. The
method of representing gender or status, for example, may have already been
decided upon, and some of the transformation rules may have already been
defined.
The transformation tool or tools used to populate the ODS will have to be able to
handle a heterogeneous environment. Most often, the legacy environment will
have data sources on many different platforms. The ODS itself may be on a
distinct platform, depending on the requirements of your situation. The tool must
be able to handle a wide range of data formats and structures. It must handle
many data types and the conversions between them. You should also choose a
tool which is constantly adding new sources and targets.
Note: An ODS is not necessarily near-real-time. It is definitely more volatile
in terms of frequency and velocity. But daily updates might meet the business
need especially in companies where monthly DW updates are the norm.
The ODS can be populated in real-time at the same instant as the operati