Hummingbird Integration Suite A Technical Overview · > Hummingbird Integration Suite — A...

Transforming Information into Intelligence™

Invoice Authorizations > Spatial Data Management > Integrated Practice Support > Matter-Centric Portal > Case Management > Critical Dates Management > Records Management > E-mail Management > Gramm-Leach-Bliley Act > IMLA-FATA > OFAC > SEC Rule 17a-4 > USA Patriot Act > DoD 5015.2 > UK PRO > MoReq > ISO 15489

> EFOIA Deal Management > Risk Management > Basel Capital Accord (Basel II) > M&A Pitch Books Management > Virtual Deal Rooms > Corporate Compliance > Correspondence Tracking > Section 508 > HIPAA > PIPEDA > E-Sign Act > GPEA > Claims Processing > Sarbanes-Oxley > Contract Management > Correspondence Logging

Conflict Management > Invoice Authorizations > Spatial Data Management > Integrated Practice Support > Matter-Centric Portal > Case Management > Critical Dates Management > Records Management > E-mail Management > Gramm-Leach-Bliley Act > IMLA-FATA > OFAC > SEC Rule 17a-4 > USA Patriot Act > DoD 5015.2 > UK PRO > MoReq

> ISO 15489 > EFOIA Deal Management > Risk Management > Basel Capital Accord (Basel II) > M&A Pitch Books Management > Virtual Deal Rooms > Corporate Compliance > Correspondence Tracking > Section 508 > HIPAA > PIPEDA > E-Sign Act > GPEA > Claims Processing > Sarbanes-Oxley > Contract Management > Correspondence Logging

> Hummingbird Integration Suite™

A Technical Overview

> Hummingbird Integration Suite — A Technical Overview

2

While every attempt has been made to ensure the accuracy and completeness of the information in this

document, some typographical or technical errors may exist. Hummingbird cannot accept responsibility

for customers’ losses resulting from the use of this document. The information contained in this document

is subject to change without notice.

This document contains proprietary information that is protected by copyright. This document, in whole

or in part, may not be photocopied, reproduced, or translated into another language without prior written

consent from Hummingbird.

This edition published October 2004


4

Table of Contents

> Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

> Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

> Data Integration for the Enterprise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Hummingbird Genio® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Hummingbird Genio Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Hummingbird Genio Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Genio Engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Genio Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Genio Designer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Genio Scheduler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Genio MetaLinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Genio DataLinks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Hummingbird Met@Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

> Hummingbird Genio Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Middleware and Network Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Incremental Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Transformation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Support of Stored Procedures and SQL Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Support for External Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Joins, Aggregation, Distinction, Filter, and Sorting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Transaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Validation Against Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Loading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Failover Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Performance Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18


5

Process Optimization and Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Parallelism and Process Slicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

> Hummingbird Genio — Unique Processing Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Use of the Genio Engine Exclusively . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Use of the Genio Engine with Processing on the Remote Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Exclusive Use of the Genio Engine with Processing on the Remote Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

> Hummingbird Genio — Superior Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Procedural Scripting Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Reusability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Tracking Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Impact Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Auto-Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

> Hummingbird Genio and Metadata Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

The Importance of Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

What is Hummingbird Met@Data? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

An Enterprise Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

> Looking Forward . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

> Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25


6

Abstract

This paper is intended for IT professionals interested in understanding and learning about Hummingbird Integration Suite.™ It begins with an introduction to the product and then it explores the product architecture and unique features, and concludes with an indication of future product developments.

Introduction

The challenge of managing and leveraging all of the data within an enterprise grows increasingly complex. More and more applications, such as Customer Relationship Management (CRM), Enterprise Resource Planning (ERP), Supply Chain Management (SCM), have become embedded in the enterprise’s daily business, and combined with Web applications and legacy systems, they have created an elaborate and complicated IT environment. Many of these applications represent large investments by the company and yet the data contained in these systems is often isolated and not easily accessible.

However, in today’s competitive and demanding business environment, organizations are recognizing the value of analyzing all enterprise data to gain a ‘single version of the truth’ — truths about customer relationships, business performance, and supplier capabilities. And, this analysis is starting to take place in ‘real time,’ as businesses operate on 24x7 requirements.

The first, and most challenging, step in this analysis process is Data Integration — accessing and consolidating the disparate data and systems to feed data warehouses, operational data stores and analytic applications — the basis for analysis of the entire enterprise. Moreover, to enable faster implementation of business processes, organizations need a solution that will provide the ability to interchange data between all systems in their IT environment.

Data Integration for the Enterprise

Hummingbird Integration Suite’s data integration solution, Hummingbird Genio® is Hummingbird’s universal data integration solution that accesses, transforms, enriches, and directs information from across the entire spectrum of enterprise systems and applications. This powerful solution spans the functional areas of data extract, load, and transformation (ETL) and enterprise application integration (EAI) to handle today’s large data volumes and ‘near real-time’ requirements. It provides accurate, consistent, and timely information by connecting any data source to any target system throughout the enterprise. Hummingbird Genio allows organizations to make more informed, strategic decisions based on a single, accurate version of the truth.


7

Hummingbird Genio

Hummingbird Genio extends an organization’s existing investment in technology and human resources by seamlessly integrating its IT infrastructure. Hummingbird Genio distributes data extraction, transformation, and load processes across computing resources and is database-independent. This allows organizations to select the RDBMS of their choice to store the Genio repository rather than locking users into a single vendor or proprietary solution.

Adding to this capability is a powerful component of Hummingbird Genio that uses native capabilities of source and target relational databases to delegate transformation processes, leveraging existing technology and minimizing network traffic.

Hummingbird Genio Architecture

The Hummingbird Genio architecture is built on an extensible, component-based, hub-and-spoke design. It features a centralized engine and metadata repository (the hub) and data sources and targets (the spokes) between which the hub exchanges data. Unlike other hub-and-spoke architecture products, Hummingbird Genio can optimize data management processes, avoid bottlenecks and reduce network traffic by leveraging the local database capabilities.

Figure 1 — Hummingbird Genio Architecture

Genio Designer

Genio Scheduler

Genio Admin Manager

Hummingbird Met@Data

Genio Repository

Genio Engine

ERP & CRMCustom

ApplicationMainframeApplication

ProgramEvents

DataWarehouse

Datamarts

Message Broker Flat Files Staging ODS Cube


8

The benefit of a hub-and-spoke architecture with a centralized and open repository is that organizations can maintain full control of all data exchange processes, business rules, and metadata that make up any and all projects within the enterprise. This enhances environment management and empowers knowledge workers to make better, more efficient use of business intelligence and analytical applications.

Since its initial development, Hummingbird Genio has followed an open and extensible design concept in order to provide a solid platform for future development. Currently, architectural extensibility has been achieved through a COM-based modularity and internal programming API (ETL Development Kit). This API provides a standardized COM API for all product functionality, thereby simplifying development of additional functionality and unifying the look and feel of different applications in the suite.

This structural architecture has enabled Hummingbird to develop a procedural approach to data transformation and exchange processing that gives users unlimited capabilities to transform and process all types of data. With this approach, users are not limited to the functions provided by the tool. Rather, they are free to develop their own re-usable code transformation to any degree of complexity.

Hummingbird Genio is built on a client/server architecture and incorporates a centralized and open metadata repository. It can be implemented within a distributed deployment model, allowing multiple developers to work on projects simultaneously with complete version control and customized access privileges.

Hummingbird Genio Components

Genio Designer

Genio Scheduler

Genio Admin Manager


Genio RepositoryRDBMS

Genio EngineNT, Solaris, AIX, HP-UX,

SuSe, Red Hat

ERP (SAP)

RDBMS Message Brokers Files

MainframesCubes

ERP (SAP)

RDBMS Message Brokers Files

Mainframes

Programs,Events

Distributed Deployment

GenioDataLinks

GenioDataLinks

Multi-developer, projectdesign environment

Windows NT/2000

Figure 2 — Hummingbird Genio Components


9

Hummingbird Genio offers an integrated set of components that allows organizations to design, deploy and maintain data transformation and exchange processes. Hummingbird Genio components include Genio Engine, Genio Repository, Genio Designer, Genio Scheduler, Genio MetaLinks, Genio DataLinks and Hummingbird Met@Data.

Genio Engine

Hummingbird Genio harnesses the power of a scalable, multi-threaded, transformation engine that brokers information from any source to any target. The Genio architecture supports distribution and synchronization of data transformation and exchange processes over multiple Genio engines. This is crucial as data volumes increase in size and as transformation processes increase in complexity. It allows Hummingbird Genio to leverage the power of existing distributed computing resources. The Genio Engine supports Windows NT® and popular UNIX platforms (Sun® Solaris,® IBM AIX and HP/UX).

Genio Repository

The Genio Repository stores and manages all aspects of data transformation and exchange process metadata. All technical metadata (e.g. data structures), business metadata (e.g. transformation rules) and production metadata (e.g. logs) are stored in this repository. This repository is database-neutral and completely open. It can reside on Oracle, Microsoft® SQL Server, Sybase Adaptive Server, IBM UDB or Informix.

Each component in a data transformation and exchange process is created as an object and stored in the repository. Since Hummingbird Genio automatically identifies any change made to metadata, it provides impact analysis, requiring every object impacted by the change to be addressed before the next data transformation and exchange process is executed. This ensures information quality and consistency.

Genio Designer

Genio Designer is a multi-user graphical environment for designing data transformation and exchange processes. Data structures can be imported directly from source and target systems or using metadata bridges (Genio MetaLinks). User-defined business rules, functions, and procedures created in Genio Designer are stored as objects within the Genio Repository and are completely reusable from project to project. Hummingbird Genio incorporates a graphical interface that provides a complete and powerful procedural scripting environment for designing data transformation processes of any complexity. Hummingbird Genio processes can be scheduled or triggered by external events, such as file modifications, file transfers and e-mails.


10

Genio Scheduler

This component programs, triggers and controls processes that can be scheduled to execute on an event or calendar basis. Genio Scheduler provides real-time control and monitoring of process executions as well as full history and audit-trail reporting. If desired, Genio Scheduler can work alongside external schedulers like IBM Tivoli® or CA-Unicenter. The substitution process is straightforward as it can be implemented using standard API calls to the underlying architecture on UNIX or Windows NT.

Genio MetaLinks

Genio MetaLinks are ‘pluggable’ metadata bridges embedded in Genio Designer that enable the import of data structures from CASE tools or ERP systems.

Hummingbird Genio currently provides the following MetaLinks:

> Hummingbird Genio MetaLink for SAP R/3 (3.x, 4.x)

> Hummingbird Genio MetaLink for SAP BW 1.2, 2.0, 3.0

> Hummingbird Genio MetaLink for SAP IDoc

> Hummingbird Genio MetaLink for Sybase PowerDesigner

> Hummingbird Genio MetaLink for Computer Associates Erwin

Figure 3 — Hummingbird Genio Designer


11

Genio DataLinks

Genio DataLinks provide native connectivity to most relational and multi-dimensional database systems and flat files. Data access is achieved through native APIs, pass-through ODBC or generic ODBC. For all connectivities, Hummingbird Genio has specific grammar that is used to generate native SQL statements. Hummingbird Genio is also able to natively access complex files such as EBCDIC or XML and allows processing of any complex files (EDI, IDoc, WebLogs, etc.).

Unlike other Genio solutions available today, Hummingbird Genio offers native population of multi-dimensional databases, such as Hyperion Essbase and Oracle Express. With this functionality, users can directly create all hierarchies or members, set all necessary attributes, and load or refresh cubes. Through native access, users do not require an additional staging area or complex multi-layer tools from multiple vendors. There are two advantages to this approach — namely much better performance due to the elimination of any staging area; and maintenance of programmatic control of multi-dimensional cubes within the transformation logic.


Hummingbird Met@Data is another key feature of Hummingbird Integration Suite. It is a metadata management solution that enables IT professionals to collect, organize and document structured and unstructured data to optimize its deployment and its use across the organization.

For more information, see the ‘Hummingbird Genio and Metadata Management’ section later in this paper.

Hummingbird Genio Features

Extraction

Hummingbird Genio extracts from the source databases using native SQL grammar making it possible to optimize the use of source database power and minimize network traffic. This is done by accessing only the source rows that are pertinent to the transformation work, thereby avoiding loading of all the data into a staging area.

When working with text sources, Hummingbird Genio has a variety of tools to help manage complex structures like hierarchical data dumps from mainframes or EDI files. Even when dealing with these kinds of files, the Hummingbird Genio function is still the same and makes no distinction if the source or target is a text file or database table.


12

Sources

File Formats

> Delimited Text files, Fixed Text files

> AS-400 Flat Files

> Standard Flat Files (Cobol)

> XML files

> SAP IDoc

> Palm OS compatible JFile, JfilePro

RDBMS

> Microsoft SQL Server, 7, 2000 (Native SQL, ODBC)

> Microsoft Access, 2000, 2002 (Native SQL, ODBC)

> Oracle 8, 8i, 9i, 10g (Native, ODBC)

> Informix 7.3x, 9.x (Native SQL, ODBC)

> Generic ODBC

> Sybase Adaptive Server 11.x, 12.x (Native, ODBC)

> Sybase IQ 12.x (Native SQL, ODBC)

> Sybase SQL Anywhere 7.x, 8.x, 9.x (Native SQL, ODBC)

> Teradata, V2R4.x, V2R5.x (Native SQL, ODBC)

> IBM UDB 7.x, 8.x (Native SQL, ODBC)

Mainframe Connectivity (Native, ODBC)

> IBM DB2 AS400

> IBM DB2 MVS, OS/930, Z/OS

> IBM IMS

> Adabas

> VSAM


> Natural

> CICS

SAP R/3 (4.x) (Native)


13

Middleware and Network Support

Hummingbird Genio supports all middleware providing ODBC, XML, file-based or message-oriented middleware (MOM), such as IBM WebSphere MQ, interfaces. By leveraging third-party solutions Hummingbird Genio is therefore able to provide access to almost any legacy system, most ERP, CRM, SCM systems and also custom applications.

Also, Hummingbird Genio can use FTP protocol, MAPI, RSH or any external application to get access to data or push data to the target environment.

Incremental Extraction

Hummingbird Genio offers multiple strategies to perform incremental extraction. Simple approaches, like selection limits based on time stamps, use of database log tables or use of database triggers to capture changes, can easily be implemented to enable incremental extraction. These techniques are environment-neutral and can be implemented without any additional software investments.

Hummingbird Genio also provides specific DataLinks that can automatically capture changes from databases or VSAM files.

Finally, Hummingbird Genio can also interface with products like Sybase Replication Server and directly process deltas created by this external middleware. This can also be done by leveraging MOM solutions or any message-based environment.

Transformation Functions

Hummingbird Genio has a complete set of transformation functions making it as capable as a programming language but providing a graphical and optimized user interface to make the work on transformation routines more productive.

Hummingbird Genio offers 100 native functions that can be used to build complex target expressions with the source data or variables. These functions cover the entire spectrum of string, dates, number or boolean manipulation. Complex clauses such as IF, THEN, ELSE or CASE can also be written in expressions. These functions are processed inside the Genio Engine on NT or UNIX, but they are translated automatically, when possible, in native SQL functions if the execution takes place on a remote database.


14

Using these standard functions, Hummingbird Genio users can create their own macros to describe business rules. For example, a function ‘Discount’ can be calculated from a given sales amount, and used everywhere across the Hummingbird Genio transformations, being processed inside the Genio Engine or on a remote database.

Functions

String Functions

Date Functions

Mathematical and

Trigonometric Functions

Numeric Functions

Miscellaneous Functions

Arithmetic Operators

Boolean Operators

Relational Operators

Additional Operators

Operators

ASCII, CHR, FILL, FORMAT (from date), FORMAT (from numbers),

LEFT, LENGTH, LOCATE, LOCATE (with start), LOWER, LTRIM, MID, MID

(with length), REPLACE, RIGHT, RTRIM, STRING (from date), STRING

(from numbers), TRIM, UPPER

ADDMONTH, DATE (from three numbers), DATE (from six numbers),

DATE (from string), DATE (from string with format), DATEADD, DATEADD

(number of days), DATEDEC, DATEDIFF, DATEDIFF (number of days),

DATEINC, DAY, DAYOFWEEK, DAYOFYEAR, HOUR, LASTDAYOFMONTH,

MINUTE, MONTH, SECOND, TODAY, WEEK, YEAR

ACOS, ASIN, ATAN, COS, COT, EXP, LOG, MOD, SIN, SQRT, TAN

ABS, CEILING, FLOOR, POWER, RND, ROUND, ROUND (with decimal),

SIGN, TRUNCATE, VAL

CASE, IFTHEN, IFTHENELSE, ISDATE, ISEMPTY, ISNULL, ISNUMERIC,

MAX, MIN, NBEXCEPTIONS, SYSVAR

+, –, *, /

NOT, AND, OR

= <>, <, >, <=, >=

BETWEEN, LIKE, NOTLIKE, Concatenation, IN, BITWISEAND, BITWISEOR, BITWISEXOR


15

Support of Stored Procedures and SQL Functions

Hummingbird Genio can also manage and use stored procedures or any piece of SQL code that can be executed on a database. These SQL commands, dependant on a particular grammar, are stored in Hummingbird Genio’s repository and guarantee the reusability of existing code.

Support for External Functions

To extend the processing capabilities of Hummingbird Genio, it is also possible to use any legacy function in a DLL written in C++ or any other language. These external functions are declared once in Genio Designer and can be used seamlessly in all Hummingbird Genio transformations, thereby preserving legacy investments.

Also, Hummingbird Genio can call any executable, external batch or shell script for specialized transformation needs.

Lastly, Hummingbird Genio can use the functions (packages, macros, etc.) defined within the relational databases, either source or target. This enables more distribution of treatments on remote databases, using their existing transformations. For example, the Oracle sequences can be reused this way.

Mapping

Hummingbird Genio provides different ways to defining mapping. Whenever possible, the tools can automatically detect mapping based on field names, field order or any custom algorithm. Also, simple graphical mapping from the source to target can be manually defined using drag and drop functionality, and more complex mapping can be done using the Hummingbird Genio graphical language.

Joins, Aggregation, Distinction, Filter and Sorting

When multiple sources, heterogeneous or otherwise, are required, users are able to define datasets (logical view on the information system) to de-normalize, join, aggregate, sort and distinguish data from the source system.

These operations are defined graphically inside Hummingbird Genio, with no need to write SQL code; nevertheless, they cover the entire functional spectrum of relational database features. One can define regular joins, external left, right joins, full outer joins, calculated joins, or recursive joins involving the same table or view several times, through aliases that Hummingbird Genio manipulates transparently. Filters are transformed in WHERE or HAVING clauses, and sorting becomes ORDER BY. Hummingbird Genio recognizes each SQL grammar, thus adapting itself to the source or target DBMS.


16

Transaction

Hummingbird Genio uses the traditional transactional mechanisms of relational databases, COMMIT/ROLLBACK, on one or more databases. Transactions can be distributed on several systems. They can occur at any moment during the execution of the interface, for each single treatment or for each important step inside a single treatment, for each functional unit. For example, it is possible to validate a transaction for each client change when processing a single table containing invoices. The COMMIT/ROLLBACK mechanisms can be triggered automatically or conditionally, based on data test results on previous interface executions. This is usually used to add failover capabilities to existing interfaces.

Hummingbird Genio can do conditional COMMIT/ROLLBACK for any module within a process. This means that Genio Designer can create a complex condition for loading and data quality checking and decide whether to commit or rollback the load.

Validation against Metadata

Hummingbird Genio is completely metadata-driven and always validates the business rules and changes against the metadata and operational sources and targets. While it is possible to write logically incorrect statements, their physical grammar is automatically tested and validated.

Loading

Hummingbird Genio has multiple loading strategies: single row, packet and bulk. In certain cases, the loading can be done by the source database directly, when the developer has decided to bypass the engine altogether.

Explained later in ‘Hummingbird Genio Unique Processing Methodology.’


17

Targets

File Formats

> Delimited Text files, Fixed Text files

> AS-400 Flat Files

> Standard Flat Files (Cobol)

> XML files

> SAP IDoc

> Palm OS compatible JFile, JfilePro

RDBMS

> Microsoft SQL Server 7, 2000 (Native SQL, ODBC)

> Microsoft Access, 2000 (Native, ODBC)

> Oracle 8, 8i, 9i, 10g (Native, ODBC)

> Informix 7.3.x, 9.x (Native SQL, ODBC)

> Generic ODBC

> Sybase Adaptive Server 11.x, 12.x (Native, ODBC)

> Sybase IQ 12.x (Native SQL, ODBC)

> Sybase SQL Anywhere 7.x, 8.x, 9.x (Native SQL, ODBC)

> Teradata V2R4.x, V2R5.x (Native SQL, ODBC)


Mainframe Connectivity (Native, ODBC)

> IBM DB2 AS400

> IBM DB2 MVS — OS/390, Z/OS

> IBM IMS

> Adabas

> VSAM

> Natural

SAP R/3 (4.x) (Native)

SAP BW (1.2, 2.0, 3.0) (Native)

Multi-dimensional Databases

> Hyperion Essbase 5.x, 6.x

> Oracle Express 5.x, 6.x


18

Scheduling

Hummingbird Genio includes a complete scheduling facility making it possible to schedule projects at a fixed time, periodically (daily, weekly, monthly), triggered by outside events or from the polling service (based on files from certain locations). Combining these functions, the Genio developer can build as complex scheduling rules as necessary. Also, the single scheduler shows all the logs for all the processes, as it is a single entry point to the whole scheduling architecture.

Genio Scheduler is not always required, as there is support for external applications setting its variables, launching processes and receiving the result of the process. This makes it very easy to implement use of system management applications like IBM Tivoli or CA Unicenter.

Failover Capabilities

Hummingbird Genio does not impose any methodology on failover functionality. The open architecture enables the developer to use any preferred technology for failover systems including “power-off” re-starts or complex rules on continuity. Hummingbird training and best practices routinely teach these different approaches and can suggest to the developer the best use for a specific case.

To permit such implementations, Hummingbird Genio provides the basic features required to implement complex failover strategy. When triggering the process execution, Genio users are able to define the list of Genio engines and timeout for each one. Within the process, users can define restart points and automate process restart.

Performance Measurement

Hummingbird Genio has a unique performance meter inside its log. All the different tasks are timed including the module coherence tests and SQL statement performance, as well as the load processes. As a result, the developer will know the transformed volume and the possible performance problems.

Hummingbird Genio can automatically e-mail this report to the developer after each execution as well as keep it in the repository, making it possible to analyze the performance.

It is also possible to use performance measurement tools to detect and isolate networks, machines or any other potential bottlenecks.

Process Optimization and Tuning

Once again, Hummingbird Genio openness delivers multiple ways of optimizing process executions.

Hummingbird Genio offers ways of offloading part of or the entire process execution on the source or the target system.

For more information, see ‘Hummingbird Genio Unique Processing Methodology’ section.

Also, Hummingbird Genio provides multiple reading and writing strategies (single, packet, and bulk) that enable Genio users to optimize data movements based on the topology of their information system.


19

When loading needs to be optimized, indexes can be dropped and restored on the target so that the database engine can accept rows at the speed they are sent from the Genio Engine. This is very simple to achieve with Hummingbird Genio’s SQL procedures and SQL functions.

Also, when required, SQL statements can be tuned using the database SQL analyzer or database hints that will ensure maximum utilization of resources. The tuning of the target is usually based on the performance of the database and the type of end-user tools used and how they are used. Typically, the database environment provides tools to assist tuning by showing statistics on the use of disk, indexes and processor time for queries.

Parallelism and Process Slicing

Hummingbird Genio has native multi-threading that enables it to load tasks in parallel; it communicates through global variables and events between the processes. Increased performance can be achieved by splitting the load work across multiple CPUs or across multiple physical servers depending on where the bottleneck is.

Hummingbird Genio has a simple facility to perform source slicing based on the RDBMS sources. Again this architecture is open and does not impose a foreign methodology on the developer. Also, Hummingbird Genio gives users an easy way of defining multiple execution contexts that will handle a subdivision of the entire process with its ‘Running Context.’

Hummingbird Genio — Unique Processing Methodology

Hummingbird Genio offers a unique methodology that distributes transformation workload by offloading certain tasks to idle database engines during off-peak hours to maximize efficiency and system performance. The following graphics and descriptions depict the three modes of transformation processing that Hummingbird Genio offers:

Use of the Genio Engine Exclusively

The first mode involves Hummingbird Genio extracting data from the source database, transforming it within its own engine, and then populating target databases directly. This architectural schema is recommended when sources and targets are heterogeneous, or when the required transformations cannot be performed by remote source or target databases.

Extract

RDBMS Hummingbird Genio Engine

RDBMS

Data Data

Transform Load

Figure 4 — Data Access Method One


20

Use of the Genio Engine with Processing on the Remote Database

The second Hummingbird Genio data access mode involves extracting data and transforming it on the source database platform. The resulting data is then transmitted through the Genio platform, and is loaded into the target database where any further transformation is carried out. If possible, aggregations are processed on the source, thereby reducing the volume of data transmitted between source and target and, as a result, the required bandwidth to move data through the network. In this mode, data is transported through Hummingbird Genio and copied without transformations to the target.

Transform/Extract

RDBMS Hummingbird Genio Engine

RDBMS

Data Data

Load/Transform

Figure 5 — Data Access Method Two

Exclusive Use of the Transformation Capabilities of Remote Databases

With the third model, the source and target are on the same server and can optimize the use of this platform. In this situation, it is not necessary for the data to leave the server or to transport the data through a communications layer. Since the Genio engine only sends SQL orders adapted to the relational database, the RDBMS manages the extractions, transformations, and insertions (or updates). As a result of this architecture, this model outperforms the two previous ones, since no network bandwidth is used.

Extract

RDBMS 1Hummingbird Genio Engine

RDBMS 2

SQL Extract/Load

Transform Transform

Figure 6 — Data Access Method Three

Each data access mode can be developed through a common Genio interface and defined using the same graphical metaphor and programming methodology. By maximizing user control over data flow, the Genio data access architecture enables users to significantly improve the performance of their data exchange processes. Being able to select, manage and summarize only relevant data, and control the platform on which work is executed will vastly improve performance.

Regardless of which data access model is chosen, Hummingbird Genio impact analysis capabilities are maintained, ensuring that, if changes are made to any element of the data exchange process, administrators are notified prior to the next scheduled execution.


21

Hummingbird Genio — Superior Productivity

Procedural Scripting Language

Hummingbird Genio provides a graphical scripting language, designed specifically for simple and complex data transformation, that supplies the transformation power typically found only in traditional 3GLs. The language permits the use of loops, tests, variables, as well as IF, ELSE, THEN and CASE statements.

Hummingbird Genio also provides high level, user-friendly commands such as SmartInsert, designed to perform row additions in tables with primary keys. SmartInsert performs an ‘insert’ by default and switches automatically to perform an ‘update’ if the row being inserted already exists in the target.

Procedural transformations can be performed on individual rows of data, on a set of rows, or on an entire table. The ability of Hummingbird Genio to manipulate data on a row-by-row basis allows the development of extremely complex transformations. The ability to support complex transformations directly within the Hummingbird Genio procedural language is critical as it permits the full power of impact analysis.

Reusability

Objects created in Genio Designer are stored in the centralized metadata repository, making them available for re-use. Objects are developed once and deployed many times, providing data integrity across the enterprise. The centralized development model of Hummingbird Genio eliminates the need to re-code business rules, lookup tables, and custom functions for each new transformation project.

Tracking Changes

Hummingbird Genio provides the ability to track differences between the state of an object stored in the Genio Repository and the state of the same object as it exists in the remote system (physical object in the source or target). Every change made to an object, whether it’s in a remote database or in Hummingbird Genio, is stored and available. The administrator has the option to update Hummingbird Genio with the current version of the object that exists in the remote database. This allows Hummingbird Genio to always be consistent with data structures, as they exist on remote sources and targets, ensuring data accuracy and consistency in every data transformation and exchange process.

Impact Analysis

Managing the status of objects in real-time allows Hummingbird Genio to perform impact analysis of changes. Genio Designer provides two impact analysis modes — immediate and deferred. In the immediate impact analysis mode, Hummingbird Genio immediately checks the impact of a change on an object and on all objects related to it. If the change impacts the integrity of an object, its status is changed to ‘Invalid Object’.


22

In the deferred impact analysis mode, when an object is modified, Hummingbird Genio changes its status to ‘Undefined Object’. Objects with this status can be verified at a later time.

Hummingbird Genio detects any modification made to a source or target data structure, eliminating the risk of data corruption or failure during subsequent transformations. For example, if a source application changes, impact analysis identifies the effect the change has on the data warehouse. Hummingbird Genio’s impact analysis eliminates the need for developers to spend time manually tracking down dependencies whenever a change is made. Hummingbird Genio provides a persistent list of invalid and undefined objects, allowing developers to know the exact state of their metadata, and the immediate consequence of making a change to it. Then this dynamic impact analysis helps developers fix impacted objects by providing a thorough description of required changes and even auto-correction mechanisms. This decreases the maintenance cycle and increases developer productivity.

Auto-Documentation

Genio Designer automatically manages the documentation of Genio projects, including dependencies between objects, modification history and comments.

Genio users can at any time automatically print or generate HTML documentation, dependency graphs or dataflow graphs.

This significantly reduces documentation efforts and ensures the accuracy of project documentation.

Hummingbird Genio and Metadata Management

With the vast amounts of information organizations have at their disposal today, there is an ever-increasing need to collect, manage and reuse data about that information — what it is, its location, its origin, or its size. This ‘data about data’ is called metadata. It can describe any characteristic of the data such as the content, its structure, its quality or any attributes related to its processing or changes. Quite simply, metadata is an important catalog of information from any number of sources — data warehouses, data exchange tools, business intelligence tools, ERP systems, business process modeling, workflow, data quality tools, document management systems or any other application dealing with data.

The Importance of Metadata

Metadata secures the lineage of data, enabling knowledge workers to gain access to business rules, and to understand where the data came from and how it has been handled to date, making the time they spend on query and analysis activity more productive.

Metadata management provides critical access for both business users and technical users working with the data. Depending on the type of user, metadata can serve either as a blueprint to the inner technical workings of the warehouse, or as a roadmap to assist in navigating the warehouse and locating useful information.


23

What is Hummingbird Met@Data™?

Hummingbird offers robust metadata functionality with Hummingbird Met@Data, delivering seamless metadata access and integration, maximizing reuse of important metadata and ensuring metadata usability. With Hummingbird Met@Data, administrators can create structured views of information and allow business users to access this information to make informed decisions, based on accurate and consistent data.

An Enterprise Approach

Hummingbird Met@Data provides a metadata management solution that can go beyond simply ETL-centric data to address decision support metadata repositories across the enterprise. Often, multiple metadata repositories exist in an organization and they are not easily integrated or synchronized. Another challenge is the interchange of metadata between back-end repositories and front-end tools like business intelligence reporting solutions. A true enterprise solution provides seamless access to all metadata sources, enabling knowledge workers to gain access to business rules and making the time they spend on query and analysis more productive.

Hummingbird Met@Data architecture incorporates standard technical metadata as well as the means to create customized classes and templates. This facilitates the use of metadata created either in ETL processes or in a customized semantic layer. By interconnecting these two elements, the Hummingbird approach to metadata provides a sound documentation tool for data warehouses and data exchange projects. Hummingbird Met@Data’s open, driver-based architecture makes it easy to implement support for other repositories. This architecture is completely open to support RDBMS repositories, as well as COM-based APIs to other vendor repositories.

Hummingbird Met@Data consists of the following components:

> Met@Data Studio is a Java-based, desktop administration tool that lets system administrators define, import, and customize classes; control access to metadata; and structure how it is presented. This means that each user sees only the information that is relevant, presented in an understandable structure that is suitable and efficient for processing the information.

> Met@Data Explorer is a fully customizable, web-based solution designed to support the information search and analytical needs of knowledge workers by providing access to all decision support metadata repositories within an organization. Met@Data Explorer allows users to browse and query the information that has been designed in Met@Data Studio.

Hummingbird Met@Data leverages a driver mechanism. Drivers communicate the contents of target repositories by delivering information about available classes and attributes. Administrators can create tree views harnessing meta-objects from different repositories and interconnect them with hyperlinks. When querying or browsing, the Met@Data Server delivers requested data to end users in the format defined in the metadata repository.

As part of Hummingbird Integration Suite, Hummingbird Met@Data is a comprehensive metadata solution that simplifies deployment, improves scalability, and reduces physical integration challenges. The resulting Web-enabled views enable business users to explore the relationships between systems, understand data transformation processes and trace the origins of their data.


24

Looking Forward

Today, the data integration market includes several types of tools. The two main categories that are used to implement data exchange are ETL (Extract, Transform, Load) and EAI (Enterprise Application Integration) tools.

ETL and EAI architectures share many common concepts; however, they address different business requirements and provide specific functionalities. Generally speaking, ETL tools are primarily batch data exchange solutions, focusing on ‘Technical’ data and working mainly with RDBMS.

On the other hand, EAI tools are considered real-time data exchange solutions, focusing on ‘Business’ data and working mainly with message-oriented middleware. However, both product categories often deal with the same data and implement the same business rules to process this data.

Today, organizations that want to implement these two types of data exchange solution typically have to use two different products, resulting in increased total cost of ownership of the data integration platform.

EAI is one logical path for expansion of Hummingbird Genio. Hummingbird already offers a strong ETL package, with some extensions into the EAI world (XML connectivity, support for IBM WebSphere MQ, etc.).

Conclusion

It would be accurate to say that data just might represent one of the last great competitive advantages for today’s businesses. If this is true, then those companies taking advantage of a robust, flexible, and fully functional data integration solution are going to be leaders in their market. Because they address the complex requirements of sophisticated IT environments, data integration solutions are at the technology forefront for many organizations. With its advanced architecture, support for the most complete range of database and hardware systems, and its near real-time functionality, Hummingbird Integration Suite offers companies around the world a best-of-breed solution for today and for the future.

Invoice Authorizations > Spatial Data Management > Integrated Practice Support > Matter-Centric Portal > Case Management > Critical Dates Management > Records Management > E-mail Management > Gramm-Leach-Bliley Act > IMLA-FATA > OFAC > SEC Rule 17a-4 > USA Patriot Act > DoD 5015.2 > UK PRO > MoReq > ISO 15489

> EFOIA Deal Management > Risk Management > Basel Capital Accord (Basel II) > M&A Pitch Books Management > Virtual Deal Rooms > Corporate Compliance > Correspondence Tracking > Section 508 > HIPAA > PIPEDA > E-Sign Act > GPEA > Claims Processing > Sarbanes-Oxley > Contract Management > Correspondence Logging

Conflict Management > Invoice Authorizations > Spatial Data Management > Integrated Practice Support > Matter-Centric Portal > Case Management > Critical Dates Management > Records Management > E-mail Management > Gramm-Leach-Bliley Act > IMLA-FATA > OFAC > SEC Rule 17a-4 > USA Patriot Act > DoD 5015.2 > UK PRO > MoReq

> ISO 15489 > EFOIA Deal Management > Risk Management > Basel Capital Accord (Basel II) > M&A Pitch Books Management > Virtual Deal Rooms > Corporate Compliance > Correspondence Tracking > Section 508 > HIPAA > PIPEDA > E-Sign Act > GPEA > Claims Processing > Sarbanes-Oxley > Contract Management > Correspondence Logging

Transforming Information into Intelligence™

Corporate Headquarters

1 Sparks Avenue, Toronto, Ontario M2H 2W1 Canada

Toll Free Canada/USA: 1 877 FLY HUMM (359 4866)

Tel: +1 416 496 2200

Fax: +1 416 496 2207

E-mail: [email protected]

For more information, visit www.hummingbird.com/wp/etl

North American Sales Offices

Boston • Chicago • Dallas • Los Angeles • Mountain View • New York

Ottawa • Raleigh • Toronto • Washington DC

International Sales Offices

Amsterdam • Brussels • Frankfurt • Geneva • London • Milan

Munich • Paris • Rome • Seoul • Singapore • Stockholm • Sydney

Tokyo • Wokingham • Zurich

WP-06-00-EN-0026.010/04

Copyright © 2004, Hummingbird Ltd. All rights reserved.

® ™ — Trademarks and logos are the intellectual property of Hummingbird Ltd.

All other company and product names are trademarks of their respective owners.

Hummingbird Integration Suite A Technical Overview · > Hummingbird Integration Suite — A...

Documents

Transcript of Hummingbird Integration Suite A Technical Overview · > Hummingbird Integration Suite — A...