IBM Mainframe-Integration Mainframe Change Data Capture

IBM Mainframe-IntegrationMainframe Change Data Capture

Presented by

The current situation:

Mainframe

Applications

+

Databases

Mobile

Applications

+

Databases

Internet/Web

Applications

+

Databases

Workstation/Client-Server

Applications

+

Databases


• ADABAS• IDMS/DB• DB/2• IMS and DL/I• VSAM• DATACOM/DB• other…

MainframeApplications

+Databases


• SQL Server• Oracle• DB/2 UDB• MS Access• other…

WorkstationClient/ServerApplications

+Databases


• Sybase• MySQL• DB/2 UDB• ORACLE• other…

Internet/Web-

Applications

+

Databases


• MS SQL• MS Access• DB/2 UDB• ORACLE• other…

MobileApplications

+Databases

The current situation – Data exchange:

Data exchange problems

• Different data formats• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information

tcVISION – Mainframe Change Data Capture

The Solution

Moving data...

→ as much as needed→ as little...→ as transparent...→ as flexible...→ as secure... AS POSSIBLE


The new data exchange generation


The tcVISION Data Capture Methods

Basic questions:

1. How current must the information be?

Data exchange between source- and target-system should be ...

• continuous / in realtime• cyclic / interval based• event based



Basic questions:

2. What are the attributes of the datasource?

The datasource attributes influence the choice for the data capture method ...

• Size of datasource• Format of datasource (Structure, DBMS)• Change frequency• Amount of changes and• available resources (CPU-power, network bandwidth)



• Usage of the DBMS logging capabilities

•IMS/DB, VSAM, DB/2, DL/I, ADABAS, IDMS, DATACOM

• Transfer of changed data in scheduled time frame

• Best for batch window

• Best for processing right after logfile creation



• Efficient transfer of entire databases

• Analyzis for data consistancy

• Best for „Initial Load“ prior to log processing

• Best for periodic mass data transfer

• One step data transfer



• Comparison of data snapshots

• Efficient transfer of changed data since last processing

• IMS/DB, DL/I, VSAM, DB/2, ADABAS, IDMS, DATACOM, Sequential files

• Flexible processing options (SORT etc.)

• Automatic creation of deltas by tcVISION



• Realtime capture of changed data

• Changes directly obtained from DBMS

• IMS/DB, VSAM, DB/2, ADABAS, IDMS, DATACOM

• Secure data storage even across DBMS restart

• Flexible propagation methods



Up-To-Dateness of changed data



Data volume



Steps:

• Extraction of database logs

• Process meta information

• Process changes of selected tables

• Propagate changes

• Implementation into target database



Steps:

• Read datasource


• Selective and structured transfer

• Implementation into target databse



Steps:

• Read datasource

• Read corresponding snapshot

• Determine changes


• Propagate changed data




Steps:

• Capture changes in DBMS

• Transfer changes into „Pool storage“

• Parallel processing of changed data from the pool

• Propagation of data to the target system



Every method has been designed to meet special needs and requirements. An efficient and secure data synchronization is guaranteed with all methods.

Summary



Transfer and Propagation

Propagation to other mainframe systems

Propagation to Internet-databases

Propagation to mobile systems

Propagation to client server databases


Transfer and Propagation

1. Propagation independent of data origin

2. Propagation to different target systems (even for the same data)


Examples

Case 1: DB/2 Log processing and implementation into DB/2 UDB


Examples

Case 2: DL/I Real-Time Capturing and implementation into a DB/2 UDB and VSAM


Examples

Case 3: IDMS- Realtime Capturing and bi-directional synchronization with ORACLE, DB2/UDB and MS-SQL


Examples

All cases are Customer implementations, that are currently in production.

SQLServer

DL/I IDMS


Examples

Steps:

1. Import of the metadata

2. Assignment of target tables

3. Definition of processing script

4. Test and implementation


Examples

Import of meta data

The tcVISION Manager publishes the layout of the log-record for every table for later use by the processing scripts


Examples

Assignment of target tables

Creation of target tablesor

Assignment of existing tables in the target system.


Examples

Definition of a processing script

Use the tcVISION script wizard to create the script

Definition of all required parameter: Input and output, LUW processing etc.


EXAMPLES

Test and implementation

Start and test the script

Implement script execution into scheduler and automatic processing


Example

Technical aspects:

Standard data exchange was not possible due to extremely large amount of data.• 760 tables• 200.000 changes per day are processed• Propagation to DB2 UDB under SUN Solaris• Processing time with tcVISION per day less than 30 minutes


Examples

Summary

Just a few steps to implement a data synchronization process

Solid monitoring and logging

High stability

Changes applied to structures of source databases are automatically processed


Examples

Steps:

1. Define a collector and pool storage +Definition of a processing script

2. Creation of target table


DL/I


Examples

Creation of a collector/pool combination for the DL/I database

Definition of a processing script


Examples

Define Collector as DBMS Extension

Define Pool to buffer the changes

Define processing script to implement into the target database

Use project wizard to specify the work-flow


Examples


Start the project using the tcVISION Control Board


Examples

Technical aspects:

Standard data exchange was not possible due to extremely large amount of data.

• 40 DLI segments• 200.000.000+ records• 150.000+ changes per day • Implementation: Realtime using DBMS Extension Capturing of changes performed by Online- and Batch-programs.


Examples

AdditionalImplementation:

If the target DB/2 is not available, changes are automatically saved into a VSAM KSDS

Highest possible recovery

Customer performs amainframe to mainframesynchronization


Examples

Steps:

Step 1:

1. Create a collector/pool combination +Definition of a processing script

2. Creation of target table


IDMS

SQLServer


Examples

Import of the SCHEMA definitions for the IDMS database

SQLServer


Examples

Define Collector as DBMS Extension

Define Pool to buffer the changes

Define processing scripts to implement the changes

Use the project wizard to define the necessary steps

SQLServer


Examples


Start the project using the tcVISION Control Board


Examples

Technical aspects:

Conventional synchronization not possible• Bi-directional updates must be supported• Updates in DB2/UDB, ORACLE and MS-SQL are captured by TRIGGERS• TRIGGERS invoke tcACCESS/ODBC

to perform IDMS updates• tcVISION recognizes updates from tcACCESS and prevents a loop-back SQLServer

Mainframe

Applications

+

Databases

Mobile

Applications

+

Databases

Internet/Web

Applications

+

Databases


Applications

+

Databases



• Different data formats• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information

Data exchange challenges:


• Automatic data conversion into the correct target format• Different data models • Large data volumes• Limited batch window• Requirement for up-to-dateness of information



• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Large data volumes• Limited batch window• Requirement for up-to-dateness of information



• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Limited batch window• Requirement for up-to-dateness of information



• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Requirement for up-to-dateness of information



• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Real-Time Capturing meets maximum requirements for up-to-date information



• Automatic data conversion into the correct target format• Adaptation of the data structure during transfer• Only transfer changed data• Batch-Compare with operational database• Real-Time Capturing meets maximum requirements for up-to-date information

The state-of-the-art demand forMainframe data exchange:


YOU will profit from:

• Efficiency• Transparency• Auditability• Up-to-dateness

www.treehouse.com

IBM Mainframe-Integration Mainframe Change Data Capture

Documents

Transcript of IBM Mainframe-Integration Mainframe Change Data Capture