v9.1.2 update

© 2013 IBM Corporation

IBM InfoSphere Information Server v9.1.2 releaseupdate and roadmap

Beate Porst ([email protected])InfoSphere Product Management


IBM Information Server: Simplified Packaging for InformationIntegration and Quality

BusinessInformationExchange

Understanding &Collaboration

• Information blueprints• Relationship discovery

across data sources• IT-to-business mapping

DataQuality

Cleansing &Monitoring

• Analysis & validation• Data cleansing• Data quality rules &

management

DataIntegration

Transformation• Massive scalability• Power for any complexity• Total traceability

Delivery• Data capture at any time• Delivery anywhere• Big data readiness

InfoSphere Information Server EnterpriseEdition:Integrating and transforming data and content to deliveraccurate, consistent, timely and complete information ona single platform unified by a common metadata layer


Mapping Platform Components to Information ServerPackages

Platform Component \ PackageBusiness

InformationExchange

Data Quality DataIntegration

EnterpriseEdition

Blueprinting and Best Practices (BPD)

Governance Dashboard

Data Discovery

Metadata Management and Lineage (MWB)

Logical and Physical Data Modeling (IDA)

Business Glossary (BG)

Data Cleansing and Enrichment (QS)

Data Quality Validation & Monitoring (IA)

Data Quality Exception Management (DQC)

SOA Deployment (ISD)

Data Specification Mapping (FT)

Extraction, transformation, load (DS)

Self-Service Data Integration (Data Click)

Change Data Delivery (CDD)


Information ServerWhat’s new in v9.1.2


TransformationProductivity Connectivity OperationsOverview Performance Admin

2010 2011 2012 2013 2014

Information Server Recent Activity

8.5 FP1

8.7 FP1

9.1

FP2 FP3

FP2

FP1 9.1.2

Data Integration Acceleration- Advanced transformation

features (looping/v.pivot)- zOS File Stage- Integrated Balanced

Optimizer capabilities

Robust Enterprise Support- New Suite Installer- Active/Passive

High Availability support- Source Code Control

Integration

Simple Data Quality- Standardization Quality

Assessment- Match Specification

Report- Match Designer Updates

Stronger Governance- Operations Console- Business Glossary Workflow- Blueprint Task Management- Metadata Asset Manager

Product Integration- Leverage Data Validation

Rules in DataStage Jobs- Advanced Data Replication

integration- Next Generation Netezza

Connectivity & Optimization- HDFS Integration

Advanced Admin & Productivity- Parallel Debugger- New Backup/restore tooling- Maintenance Mode- Stronger Encryption

Agile integration- InfoSphere Data Click- Enhanced Workload Mgmt- ODM Integration- Hadoop Balanced Optimization- HDFS Extensions- InfoSphere Streams Integration

Business Driven Governance- Policy and rules support for

information governance- Web-based blueprints- Integrated metadata mgmt

enhancements

Sustainable Quality- Data Quality Console- Standardization Rules Designer- Data Rules Advancements

- IDA 8.5 support

Anywhere Integration- Big Data Features:

* JSON support*BDFS REST API*JDBC connector

- DB2 on z/OS load optimization- Data Click new data

sources/targets

Sustainable Quality- New QS standardization rulesets

(Thailand , Ireland , update forIndia)

- DQ Exception Mgt for DS/QS- Operational DQ Rules

Business Driven Governance- Bulk metadata import- Governance Dashboard- IDA 8.5 support4


Data IntegrationData Governance InfrastructureOverview Data Quality

5

Information Governance Dashboard

What’s New in Information Server v9.1.2

Usage• Raises data confidence with

visual governance status

Value• Immediate insight into

governance policy status• Interception of issues when they

start, right at the source• Effectively measure results &

compliance of policyenforcement

1000sof data pointsand policiesvisualized



6

Support for Information Data Architect 8.5

• Builds on the new metabroker introduced at 9.1 for Information Data Architect which:• introduced better performance at lower resource cost• removed Windows only dependency

• Certification of IDA v 8.5 added• Tolerance for orphaned and invalid objects (ability to ignore those that don’t impact rest of

model)• Improved error/warning logging




7

Metadata Workbench Enhancements

JDBC Connector support• Display details for JDBC Connector stage, including URL Definition, Schema, Table and SQL

statements• Includes inclusion of JDBC (writes/reads) into data lineage flowsXML/JSON Support• Browse, query and detail display for XML/JSON• Displays column level information within asset page• Can be linked via manual binding for lineage




8

Business Glossary Enhancements

Integration with Data Rules• Data rule asset types (including unpublished rules) from InfoSphere Information Analyzer are

now displayed in the Browse All Assets page, can be searched, assigned to terms, governancerules, business labels and data stewards

• Drill down from a GovernanceRule to a Data Rule to theDatabase column to which itsapplied




9

Business Glossary Enhancements

Workflow Roles• Development Log now captures every history event including creation and reviewer comments• Security roles have been changed to provide a higher degree of granularity for existing roles:

Author, Published and Reader• Two new workflow roles:

• Reviewer: can review changes and make comments• Approver: can approve changes to a new or existing term (but no edit abilities themselves)

• Can now add comments at every stage of the workflow process.

Export Development Glossary• Can now export either development or published glossary




10

Data Quality Exception Management for DataStage and QualityStage


Usage• Collect exception data from any Data

Integration or Data Quality process• Support clerical review• Data Steward Dashboard

Value• Promote consistency in the way data

stewards and business analysts caninvestigate data issues

• Insert good data quality controls andgovernance practices into eachproject

• Support a variety of processingmechanisms at the point of greatestefficiency



11

InfoSphere QualityStage Standardization Rules

New Standardization Rules• Country specific rule sets for India, Ireland and Thailand• Provide for data standardization of names (individual and organizational), addresses, phone and

locality (varies per country)• Delivered as archive files in the QSRules folder of the install directory

• Client = ./InformationServer/Clients/Classic/QSRules• Server = ./InformationServer/Server/PXEngine/QSRules




12

JSON Document Support

• Derive metadata format automaticallyfrom sample JSON documents…

• Supports hierarchical formats withsimple fields, objects and arrays

• Schema views• New Parsing and Composing steps for

provide for complex hierarchical data inJSON syntax; with value and structurevalidation options

• Multiple options for reading/writing data:- files directly from disk- as part of a long string- passed in/out as a LOB




13

JDBC Connector

• JDBC Connector provides InformationServer products with access to JDBC datasources

• Supports data read and write operationsand metadata import operations

• Certified in this release with ApacheDerby and IBM Big Insights Big SQLdrivers

• Managed metadata import providedthrough new capabilities in InfoSphereMetadata Asset Manager (IMAM)

• Filtering by asset type and name patterns




14

DB2 Z Bulk Load Optimization

Huge Performance Gains For Load• Moved from a single load stream to parallel streaming via Z pipes.

• Multiple LOAD utilities targeting separate partitions which performsfaster than a single LOAD utility targeting all partitions.

• 9.1.2 is 80 to 160% faster than 9.1.0 (depending on number of partitions.)

• Performance scales almost linearly as you increase the number of partitions, regardless of load method.

• Internal testing loading almost 1TB per hour using 16-way load

Huge Performance Gains For Read• connector determines the number of partitions in the table and dynamically configures the number of

DataStage nodes to match the number of partitions

• Parallel read using the 9.1.2 DB2 connector is 40% faster than the 9.1.2 DB2Z stage, regardless of thenumber of partitions.

Resilience• When Retry on connection failure is set to Yes the connector will try to establish an FTP connection again

when the initial attempt to connect fails.




15 IBM CONFIDENTIAL


Overview

• Business users need quick and easy access to information to supporttheir analytical projects.

• Organizations need to avoid data sprawl, so governance best practicesmust be ensured

• Originally only supported DB2 or Oracle to Netezza

New in this release

• Universal Connectivity via ODBC to now support DB2, Netezza, Oracle, Teradata, Sybase, SQL Server,Greenplum, and others…. as source or target

• Automatic filtering of columns with data types not supported by the target data store

• Leverages connector framework enhancement for data sampling via “row limits”

http://www.youtube.com/watch?v=hUGGudh2iWI&feature=youtu.be

InfoSphere Data Click - Self Service Data Integration



16

Connector Enhancements

Limit number of returned rows• New property to support database sampling(required for feature of Data Click)• Applies to the following Connectors: ODBC, DB2, Netezza, Oracle, Teradata and JDBC

ODBC Connector expanded binary support• The ODBC Connector now supports automatically generated 'CREATE TABLE' statements for

types Binary, VarBinary or LongVarBinary


Name Label Description Default value

LimitRows Limit number ofreturned rows.

Select Yes to limit the number of rowsthat are returned by the connector.

False (No)

Limit Limit Enter the maximum number of rowsthat will be returned by the connector.

1000



17

InfoSphere Metadata Asset Manager - Performance Optimization


• Performance benefits of BI Simplification in 9.1• 46% reduction in execution time of Express Import

• Performance benefits of physical model import (Erwin)• 44-60% reduction in execution time of Express Import (Erwin)

• IMAM Express Import in 9.1 is 7 - 1200% faster than in 8.7 for the following workloads• IDA:

• Small workload (55K assets): +1200% (Throughput: 450 objects/s)• Large workload (119K – 430K): 9.1 succeeded (Throughput: 245 – 367 objects) whereas 8.7 failed

due to Out of Memory• Erwin (124K assets): +50% (Throughput: 351 objects/s)• BO import (175K assets): +18% (Throughput: 318 objects/s)• DB2 PDR (108K assets): +7% (Throughput: 149 objects/s)• Cognos (141K assets): -20% (Throughput:120 objects/s)

• MITI in 9.1 extracts more metadata (+45% reports &+27x more models, etc) than MITI in 8.7

Note: Performance results may vary in other environments



18

Connectivity Accelerator


• Growing number of pre-build Connectivity Sample• Cassandra• Hive• Hbase• MongoDB• Avro• Jaql• JMS• WTX• and more…

https://www.ibm.com/developerworks/community/files/app?lang=en#/folder/4645e12a-7bdb-40ed-a103-f1160b707758?sort=collected


Thank you

v9.1.2 update

Data & Analytics

Transcript of v9.1.2 update