Benchmarking the SSIS connectors for Oracle, DB2 and SAP

Franck SidiTechnology Architect SQL & BIMicrosoft Israel

Franck SidiMicrosoft – 2004 to Now

Microsoft France – SQL Server PreSales Technical LeaderMigrations Specialist

Oracle, Sybase, DB2SAP Integration, BI Architecture

ATG – 2000 to 2004ATG France – PreSales Technical ManagerKeywords : Java, J2EE, Portal, eCommerce, Search, ECM, Mainframe Integration

Sybase – 1994 – 2000 SYBASE INC USA 1999 – 2000 – Senior Developer in charge of Peoplesoft Inc and Kenan (Telco Billing ISV)Sybase France 1994 – 1999 – Tech Support Engineer & Presales Benchmark Specialist (Sybase, Sybase IQ)

1993 - Master Degree Database Systems (6 Years at University)

Learn on Teradata & Oracle

Session ObjectivesSession Objective(s):

Give overview of the connectivity capabilities with SSIS for Enterprise databases and solutions

Oracle, DB2 and SAP

Give some quick feedback about Performance capabilities with Oracle DriversTips and Tricks

Data Flow Connectivity Options

Data Flow Task

OLEDBConnector

ADO.NETConnector

CustomConnector

OLEDBProvider

NativeADO.NETAdapter

ADO.NETODBC Data Source

SpecificInterface/Protocol

Data Source Specific Interface/ Protocol

ODBC

Data Storage

SSIS Connectivity for Enterprise Data Versions Suggested Connector SSIS

Source & Destination

SSIS Connection Manager

SQL Server All SQL Data Access

ComponentsADO.NETOLEDB

ADO.NETOLEDB

Oracle 9iR2, 10, 11 Microsoft Connector for Oracle

Oracle MSORA

Teradata V2R6.0/1/2, 12

Microsoft Connector for Teradata

Teradata MSTERA

SAP BI 7 Microsoft Connector for SAP BI

SAP BI SAPBI

DB2 8.1, 9.x Microsoft OLEDB Provider for DB2

OLEDB OLEDB

SAP R/3 4.6c, 4.7, 5, 6

BizTalk Adapter 3.0 for mySAP

ADO.NET ADO.NET

Siebel 7.5, 7.7, 7.8, 8

BizTalk Adapter 3.0 for Siebel

ADO.NET ADO.NET

DB2

Microsoft OLEDB Provider for DB2Interactive and scriptable Setup programSNA Trace Utility and Trace ViewerTCP/IP network connectionExecution of dynamic SQL commands (DDL and DML), including CALL statement for stored proceduresCustomized Data Link property dialog boxes for creating and modifying file-persisted OLE DB data link files.Supports Integration Services, Replication, Analysis Services, Reporting Services, DQP

Packaging and licensingInclude with Microsoft SQL Server 2008 Enterprise and Developer editionsFeature Pack for SQL Server 2008Microsoft download site

Extracting from DB2 / AS400Objectives:- Extract a single Table with 80 columns and analyze the

throughput

Results:

• 19 503 Rows / Sec extract from the Development Interface : BI Studio

• Almost 3 Times faster than the current solution XXXX

• 70% faster than YYYYY

• Parameters to check :

• Defer Prepare to True

• Rowset Cache Size

• Work with small Buffers : 1 000 Rows

Loading inside DB2 / AS400Objectives:- Load inside DB2 / AS400 - Use Microsoft OLEDB Driver for DB2 V3 (Pre-release)

- V3 includes Bulk Load Capabilities to DB2 / AS400

• Results:

• 1 000 Rows / Sec with a single load (Normal Mode)to DB2 / AS400

• 12 000 rows / Sec in single load in Bulk Mode to DB2 / AS400

• 50 000 rows / Sec in parallel load in Bulk Mode to DB2 / AS 400

TransformationObjectives:- Extract 10 336 125

rows from DB2 / AS400 using a single query

- Execute 4 Lookups transformation on DB2 to add lookup columns

- Execute Global aggregation and generate a flat file with 7 773 702 rows

Results:• Execute the global

package in 2 Minutes and 24 sec

• Execute 4 lookups on DB2 / AS400 Tables

• 54 000 Rows / Sec Throughput

• 4 Times Faster than the current solution : XXXX

SAP

Extract from SAP

Microsoft Connectors SAP R/3 (MYSAP) et SAP BW

Tables, Query, BAPI, RFC

Partner : Theobald Xtract IS

Table (Transparent Tables SAP)Query, BAPI, RFCDelta-QExtract from ABAP Programs

Microsoft Connector

Theobald Connector

•Real transaction-secure delta transfer of data

•Support of all master and variable data including own generic extractors

•No installation in SAP required

•All business contents are available immediately even though no application-specific knowledge

Oracle Connectivity with Attunity Connectors

Where Attunity Connector for Oracle Fits

Wherever there is a need for extracting or loading large volumes of Oracle data!

Oracle Connector by Attunity

Architecture

Oracle

DB

SSIS

Oracle Client

OCI

Oracle Connector by Attunity

Supported platforms & versions

Oracle

DB

Any platform

SSIS

Windows XP x86, x64Windows Vista x86, x64Windows Server 2003 x86, x64, IA64Windows Server 2008 x86, x64, IA64

Oracle Client

SQL Server 2008

9.2.0.4 and higher

RequirementsOracle client software version 10.x or 11.x, on the computer with SSIS. Use 32-bit client on 32 bit platforms, 64-bit on 64-bit platformIf using a 64-bit platforms as a designer client (i.e.; using BIDS) both 64 and 32-bit clients should be installed.

SQL Server 2008 - Enterprise or Developer Edition

Windows 2003,2008

Oracle Connector by Attunity

High Performance ArchitectureOracle integration (using OCI):

1. Bulk extract source using OCI Array Binding

2. Bulk load destination using OCI Direct Path (fastest,

constraints must turned off)

3. Incremental Bulk load using OCI Array Binding

4. Oracle Connection Manager

SSIS integration:

Direct integration into internal buffering APIs, cutting

through .NET and other layers

Oracle

DB

SSIS

Oracle

Client

OCI

Oracle connector test setupA generous two-machine configuration

Simple SSIS tasks!

Attunity Oracle connectors

SSIS (SQL Server 2008)

Windows Server 2008

Oracle 11g

Windows Server 2008

Flat files

Oracle DB

1 GB Etherne

t

24 cores @ 2.4 GHz32 GB RAMHP disk array, partitioned as 17 logical drives64-bit software

24 cores @ 2.4 GHz32 GB RAMHP disk array, partitioned as 17 logical drives64-bit software

Oracle connector test setupOracle DDL

“Mixed data types”CREATE TABLE LINEITEM (

L_SHIPDATE DATENOT NULL, L_ORDERKEY NUMBER(19,0) NOT NULL, L_DISCOUNT NUMBER(10,4) NOT NULL, L_EXTENDEDPRICE NUMBER(19,4) NOT NULL, L_SUPPKEY NUMBER(10,0) NOT NULL, L_QUANTITY NUMBER(5,0) NOT NULL, L_RETURNFLAG CHAR(1) NOT NULL, L_PARTKEY NUMBER(10,0) NOT NULL, L_LINESTATUS CHAR(1) NOT NULL, L_TAX NUMBER(10,4) NOT NULL, L_COMMITDATE DATENOT NULL, L_RECEIPTDATE DATENOT NULL, L_SHIPMODE VARCHAR2(10) NOT NULL, L_LINENUMBER NUMBER(10,0) NOT NULL, L_SHIPINSTRUCT VARCHAR2(25) NOT NULL, L_COMMENT VARCHAR2(44) NOT NULL

)

“String data types”

CREATE TABLE LINEITEM ( L_SHIPDATE VARCHAR2(10),

L_ORDERKEY VARCHAR2(10),

L_DISCOUNT VARCHAR2(10),

L_EXTENDEDPRICE VARCHAR2(10),

L_SUPPKEY VARCHAR2(10),

L_QUANTITY VARCHAR2(10),

L_RETURNFLAG VARCHAR2(1),

L_PARTKEY VARCHAR2(10),

L_LINESTATUS VARCHAR2(1),

L_TAX VARCHAR2(10),

L_COMMITDATE VARCHAR2(10),

L_RECEIPTDATE VARCHAR2(10),

L_SHIPMODE VARCHAR2(10),

L_LINENUMBER VARCHAR2(10),

L_SHIPINSTRUCT VARCHAR2(25),

L_COMMENT VARCHAR2(44)

)

Test definitionsEmptyTable

SSIS reads from flat files, writes into OracleTable is initially emptyAttunity connector Fast Load calls Oracle DirectPath API

AddToTableSSIS reads from flat files, writes into OracleTable already has data in it

ExtractSSIS reads from Oracle, writes flat files

Test measurementsMeasured throughput (rows/sec)Multiply by 133 bytes/row to get bytes/sec

Mixed data types vs. string-only

Can we exploit this difference?

mixed

CPU usageFast Load

Not Fast Load

SSIS ProcessOracle

Process

Local vs. remote execution

Difference is generally not significant

Local Remote

mixed - local

mixed - remote

string - localstring - remote

32-bit vs. 64-bit performance

Probably not enough to worry about

64-bit 32-bit

Moving data from Oracle to SQL Server

Face it, we want to move data out of OracleString data extracts are faster than mixed typesReal-world data has various data typesHow to extract as fast as possible?

1. Keep mixed types in Oracle2. Extract strings by converting in query3. Convert to SQL Server types in SSIS data

flow

Keep mixed types;

Extract stringsSELECT TO_CHAR(L_SHIPDATE), TO_CHAR(L_ORDERKEY), TO_CHAR(L_DISCOUNT), TO_CHAR(L_EXTENDEDPRICE), TO_CHAR(L_SUPPKEY), TO_CHAR(L_QUANTITY), L_RETURNFLAG, TO_CHAR(L_PARTKEY), L_LINESTATUS, TO_CHAR(L_TAX), TO_CHAR(L_COMMITDATE), TO_CHAR(L_RECEIPTDATE), L_SHIPMODE, TO_CHAR(L_LINENUMBER), L_SHIPINSTRUCT, L_COMMENTFROM ATTUSER.LINEITEM

Convert in SSIS

data flow

Two conversions are better than one? Really?

Extract from Oracle with Attunity connectorInsert using SQL Server connector

Tips and Tricks

Parallel ModeWhat does the SSIS Oracle Destination Parallel property mean in Fast Load mode?

The Parallel Load option is available in the Oracle Destination component when choosing the Fast Load mode.

This setting tells Oracle not to lock the target table exclusively for this Fast Load session. This allows running additional fast load destination components to load the same target table in parallel.

Does this mean that the Oracle Destination component will load the target table in parallel sessions?

No, it means that the table will not be locked exclusively by the Oracle Destination therefore the user can design multiple data flow tasks that loads the same table.Here is an example of two tasks loading the same table: A Source component reads records 1-500 and the Fast Load Destination component loads them to the target Oracle table.A Source component reads records 501-1000 and the Fast Load Destination component loads them to the target Oracle table in parallel.Note that both Destination components should have the Parallel property checked.

Support : http://www.attunity.com/forums/microsoft-ssis-connectors-attunity/

Error Message: Failed to load OCI DLL

SUMMARY

The error indicates a failure when loading the Oracle oci.dll. The error could occur when having multiple Oracle homes or permission problems.

SCENARIO 1

The user is working on Microsoft Windows 64Bit operating system and have installed the following components: Oracle Client 32Bit (Required for Design) Oracle Client 64Bit Microsoft SSIS Oracle Connector 64Bit This is the required list of components for Developing on 64Bit Windows machine.

On some machines the following error was reported when running the Oracle component in 32Bit (Design time):

Error at Package [Connection manager "Oracle Connector 1"]: Failed to load OCI DLL.SOLUTION

First check that the environments are working and configured properly, and that SQLPlus is responding on both installations of the Oracle Client.

On the machine which we encountered the error, it was noticed that the Oracle Home for the 64Bit Oracle Client was defined in the 32Bit registry portion (Wow6432Node), this was causing the problem of loading the wrong oci.dll.

To workaround the problem, the user should define a dummy registry entry (Z_SSIS) as follows: 1. Open the regedit utility.2. Locate the following Key: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ORACLE.

ssis_oracle1.png3. Right-Click on ORACLE node and click on New -> Key.4. Call the new key Z_SSIS (to make sure it's the last entry).5. Right-Click on the Z_SSIS node and click on New -> String.6. Name the property ORACLE_HOME.7. Double-Click on the ORACLE_HOME and set it to the location of the Oracle 32Bit installation home directory.

Attunity Oracle-CDC for SSIS

34

1. Minimal impact on Oracle (log-based CDC, no install on Oracle)

2. Efficient and reduced use or resources (process only changes)

3. Fully integrated into SSIS (2005, 2008)

4. Synchronizes Bulk-Load and CDC5. Monitoring & control console

Complete solution for integrating changes made to Oracle tables using SSIS, efficiently and in real-time

Oracle SQL Server

Change Stream

log

SSIS

Additional Resources Download Connectors

Oracle SSIS Connector

http://www.microsoft.com/downloads/details.aspx?FamilyID=d9cb21fe-32e9-4d

34-a381-6f9231d84f1e&DisplayLang=en

Teradata Connector

http://www.microsoft.com/downloads/details.aspx?FamilyID=d9cb21fe-32e9-4d

34-a381-6f9231d84f1e&DisplayLang=en

DB2 Connector

http://

download.microsoft.com/download/0/E/6/0E67502A-22B4-4C47-92D3-0

D223F117190/DB2OLEDB.exe

SAP BI

http://

www.microsoft.com/DownLoads/details.aspx?familyid=B33D2C78-105

9-4CE2-B80D-2343C099BCB4&displaylang=en

SAP Theobald

http://www.theobald-software.com/cms/en/haupt/evaluation.html

Others Links

http://www.codeplex.com/ssisctc35

http://www.codeplex.com/ssisctc

Questions