Ibm Optim Techical Overview 01282009

© 2009 IBM CorporationApril 12, 2023

IBM Optim Technical Overview

Charles LucasIBM Data Management Specialist

Information Management – Optim

© 2009 IBM Corporation2

Agenda

Challenges

Optim Concepts

Optim Architecture

Optim for Archiving

Optim Test Data Management

Optim Data Privacy

Summary


Challenges



Challenges Facing Customers Today

Mitigate Risk– Effectively and securely manage archived data– Protect data privacy– Accurate, prompt responses to auditing requests

Maintain Performance in face of Data Growth– Improve application performance by moving historical transaction

records to a safe, secure archive– Achieve Service Level Agreements (SLAs) consistently

Control Costs– Reduce infrastructure costs; utilize cost effective tiered storage– Minimize cost and time for compliance– Improve productivity of development team



Challenges: Reduce Risk Insiders and hackers are targeting data for profit

Data in and of itself has monetary value:

– Credit Card Number With PIN - $500– Drivers License - $150– Birth Certificate - $150 Source: USA TODAY research 10/06

Average cost of a data breach in 2007 was $197 USD per customer record leaked Source: Ponemon Institute

This has been a driving factor for creating data protection and privacy regulations

How to protect Personal Identifiable Information (PII)?



Challenges: Dealing with Data Growth

Data is growing at a very rapid rate – Annual growth rates for databases exceed 125%– Mergers & acquisitions– Data Warehousing

The Data Multiplier effect – OLAP cubes, data marts, and so on– Copies of data for test, development, quality assurance,

disaster recovery, etc.

Retention of data for compliance purposes

Info 2.0 applications are verbose

How to manage data growth and aging effectively?



Challenges: Control Costs

Growing storage costs due to rapid data growth

Cost of storing and managing the many copies of your production data

Cost of implementing data privacy measures for compliance across different databases and applications

Cost of archive retrievals for compliance requests and e-discovery

Growing development time costs

How to control storage and data management costs?


Optim Concepts



Optim Concepts

Complete Business Object

Federated Data

Enterprise Architecture

Terminology



Complete Business Object

Referentially-intact subset of data across related tables and applications; includes metadata

Provides “historical reference snapshot” of business activity Federated object support across enterprise data stores

PaymentsPayments



Retek / Oracle Other apps / any DBMS

Custom Inventory Management / DB2

Capture related business objects and

processes from across the enterprise

Federated Data Support



Enterprise Architecture

Platform independent architecture acts as central point to extract, store, restore and transform application data.



Optim Terminology

Referentially-Complete Optim extracts data based on primary/foreign key

relationships (“parent/child”) between tables. Handling data this way reduces errors and allows data to be

moved without breaking application software

Subsetting Using Optim to create a reduced size but referentially

complete copy of a database for development or test.

Masking Changing sensitive data before testing by replacing it with

false but equally valid data.


Optim Architecture



Optim Architecture

Optim Workstation – Optim installed on a Windows PC capable of performing all

Optim functions directly against a data source or by connecting to an Optim Server.

Optim Server– Optim installed on a Unix or Windows server that handles

requests from Optim Workstations or the command line.

Open Data Manager (ODM)– Allows access to archive files as an ODBC data source.– Allows access to archived tables through Oracle Heterogeneous

Services as a Database Link



Architecture

Optim Architecture

Optim Workstation

Optim Server

EnterpriseReporting

Tools

ODM

Optim Universal Database Access Layer

ERPCRMCustom

AppOptim

Directory

Archive

Archive

Archive

Archive

Dev QA



More Important Terminology

Optim Directory– A DBMS-based repository for Optim metadata

DB Alias– How Optim refers to a database

Access Definition– A description of what to extract and how to extract it

Relationship– User defined connection between the data in two tables based

on matching one to many columns in each table

Primary Key– User defined set of columns unique within a table



The Optim Directory

An “instance” of Optim

Holds all Optim metadata in database tables

Security possible using windows domain/user

Maintains a directory of archive Files

Created by Optim configuration when “Configure the First Workstation” is performed

Additional users can get access by using the “Configure Additional Workstation” dialog



Database Alias

Defines database in an Optim directory.

Created by Optim Configuration

Allows any database object to be addressed and accessed by a 3-part address:– DB Alias– Owner– Object Name

Allows consistent naming across the enterprise

More than just a connect string!



Created in Optim Design GUI

Defines a set of tables and relationships which can be traversed relationally

Archive Actions, which are stored procedures or DML statements, may be fired at key points of control

Every extract, delete or restore requires an Access Definition– These may be named and shared or local and tied to a

single job.

Access Definition



Relationships are automatically found when primary keys and foreign keys are defined in DB.

User-defined primary keys and relationships can be created in Optim in GUI designer or imported.

Relationship can be cross-database (between two databases, named by DB Alias)

Relationships

OptimDirectory

DatabaseAlias

Primary Keys

Relationships


Optim For Archiving



Archive, Retention and e-Discovery

Production

Extract

Restore

Archive

E-Discovery

Universal Access to Data Optim safely moves

inactive or historical data to an archive

Archive can be accessed in many ways



Production

Selective RestoreCurrent

HistoricalRestored

Archives

Reporting Data

Historical Data

Reference Data

Archive

Complete Business Object is historical snapshot of activity Storage device independence enables ILM Immutable file format enables data retention compliance

Universal Access to Application Data

Application Application XML ODBC / JDBC

Optim Data Growth Solution: Archiving



Why Optim Test Data Management?

Improve development productivity– Faster turnaround time– Supports test automation– Easier to create/verify test results– Multiple sandboxes

Better quality data (more frequent refreshes)

Control costs

Reduce storage per test instance

Ease DBA workload



Test Data Management

Easily maintain test environments

Create targeted, “right-sized” subsets faster and more efficiently than cloning

Production

DevQA

Test

LoadInsert / Update

Compare

ExtractFiles

Extract




Data Fixes

Compare Results

TEST

Go Live

ProductionApplication

Refine Data

Copy Production Data for Testing

Refresh Test Data

Optim Extract

Optim Edit

Optim Compare

Optim Extract

Optim Edit



FIND CUSTOMERNOTE INFOEXIT TABLE

FIND ORDERSNOTE INFOEXIT TABLE

FIND DETAILSNOTE INFOEXIT TABLE

Single Table Editors The Relational Editor

CUSTOMERS

ORDERS

DETAILS

........................ ........................ ........................ ........................ ........................

Traditional vs. Relational Tools

One table or view at a time No edit of related data from

multiple

Simultaneous browse/edit of related data from multiple tables and databases

Insert, delete, update Audit trail, access controls



Optim Compare

Find unexpected changes (or validate expected changes) For application testing, QA, and to verify database

contents Single-table or multi-table compare Creates compare file and/or compare Report of results

Master Copy

Latest Test

Files

Reports

Compare


Optim Data Privacy



Why IBM Data Privacy?

Protecting sensitive data

Regulatory & Compliance – PCI – HIPPA– EU Safe Harbour– Many more…

Off shoring testing

Sub subcontracting test & dev.

Good business practice

Training environments



Optim Data Privacy Solution

Production

Contextual,Application- Aware,

Persistent Data Masking

Contextual,Application- Aware,

Persistent Data Masking

EBS / Oracle

Custom / Sybase

Siebel / DB2

Test

EBS / Oracle

Custom / Sybase

Siebel / DB2

Substitute confidential information with fictionalized data Deploy multiple masking algorithms Provide consistency across environments and iterations Enable off-shore testing Protect private data in non-production environments



Data Privacy

Production

Transform and Mask Masked

Test Data

A comprehensive set of data masking techniques to de-identify data

Replaces (masks) confidential data with contextually accurate but fictionalized data



De-Identify test data

Can Be Performed

– During Extract Process from DB– During Insert/Load Process to DB– Or as a Standalone Convert Process

Transform or mask sensitive data using :

– Standard rules: Literals, Special Registers, Expressions, Default Values, Look-up tables

– Intelligent transformation rules: PCI, Addresses etc.– Custom mapping rules: user exits

Converted extract file is safe to share – sanitized data



Masked fields are consistent

Data is masked

SSN#s

157342266

132009824

SS#s

157342266

132009824

DB2

SSN#s

134235489

323457245

SSN#s

134235489

323457245

Client Billing Application

Consistent mapping Across the enterprise



Map unlike column names Transform/mask sensitive data Datatype conversions Column-level semantic date aging Literals Registers Calculations Default values Exits

Social Security (US) Credit Card Email Hash Lookup Lookup Random Lookup NAME tables (US)ADDRESS table (US) Shuffle String manipulationCurrency conversion

Masking Functions



Example: Bank Account Numbers

First Financial Bank’s account numbers are formatted “123-4567” with the first three digits representing the type of account (checking, savings, or money market) and last four digits representing customer ID

To mask account numbers for testing, use the actual first three digits, plus a sequential four-digit number

The result is a fictionalized account number with a valid format:

–“001-9898” becomes “001-1000”–“001-4570” becomes “001-1001”



Example: First and Last Name

Direct Response Marketing, Inc. is testing its order fulfillment system

Fictionalize customer names to pull first and last names randomly from the Customer Information table:

–“Gerard Depardieu” becomes “Ronald Smith”

–“Lucille Ball” becomes “Elena Wu”

Optim ships with over 5,000 male/female names and over 80,000 last names



Example: Addresses

Direct Response Marketing, Inc. is testing its order fulfillment system

Fictionalize customer addresses to pull an entire address from the

Customer Information table: “111 Campus Drive Princeton, NJ

08540 ” becomes…

“1223 E. 12th Street NY, NY 10079”

Optim ships with over 100,000 valid CASS addresses



Example: Intelligence

Generating valid social security numbers (as defined by the US Social Security Administration)

Generate valid credit card numbers (as defined by credit card issuers)

Generate desensitized e-mail addresses– Generate Email address based

on format: name@domain



Using Custom Masking Exits

Apply complex data transformation algorithms and populate the resulting value to the destination column

Selectively include or exclude rows and apply logic to the masking process

Valuable where the desired transformation is beyond the scope of supplied Column Map functions

Example: Generate a value for CUST_ID based on customer location, average account balance, and volume of transaction activity


Summary



Summary IBM Optim helps solve 3 major challenges for enterprises

today:

– Migrate Risks– Maintain performance in the face of major data growth– Reduce Costs

IBM Optim enables effective ILM

The IBM Data Growth solution keeps high performance of applications in the face of data growth by archiving inactive data

Once archived, it supports prompt, accurate responses to audit and discovery requests



Summary

Test data management can speed delivery of developed applications

IBM Optim’s data masking capabilities protect privacy by de-identifying sensitive data

Pre-built modules for many popular applications are supported by IBM Optim

Optim is a recognized market leader and used successfully by customers in almost all industries

Ibm Optim Techical Overview 01282009

Technology

Transcript of Ibm Optim Techical Overview 01282009