SOA and WebServicesEnablement of WebSphere Transformation Extender … · SOA and...
Transcript of SOA and WebServicesEnablement of WebSphere Transformation Extender … · SOA and...
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
SOA and WebServices Enablement of WebSphereTransformation Extender Maps Using WebSphereMessage Broker
WebSphere Software
2
Agenda
�WebSphere Message Broker highlights
�WebSphere Transformation Extender (TX) Product Overview
�Developing a Solution Using TX
�WebSphere TX for Message Broker
�Use case: WebService enablement of a TX map using WebSphere Message Broker
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
WebSphere Message Broker Highlights
WebSphere Software
4
Apps &
Info Assets
Business Innovation & Optimization Services
Development
Services
Interaction Services Process Services Information Services
Partner Services Business App Services Access Services
Integrated
environment
for design
and creation
of solution
assets
Manage and
secure
services,
applications
&
resources
Facilitates better decision-making with
real-time business information
Enables collaboration
between people,
processes & information
Orchestrate and
automate business
processes
Manages diverse data
and content in a
unified manner
Connect with trading
partners
Build on a robust,
scaleable, and secure
services environment
Facilitates interactions
with existing information
and application assets
ESBFacilitates communication between services
IT Service
Management
Infrastructure Services
Optimizes throughput, availability
and performance
WebSphere Process Server
WebSphere PortalWebSphere Information
Server
WebSphere Application Server
WebSphere Partner Gateway
WebSphere Adapters
WebSphere XD
WebSphere Message Broker
WebSphereTransformation
Extender for Message Broker
IBM SOA reference Architecture
ENTERPRISE SERVICE BUS
WebSphere Software
5
What is an Enterprise Service Bus?
An Enterprise Service Bus (ESB) is a flexible connectivity infrastructure for integrating applications and services.
An ESB powers your SOA by reducing the number, size, and complexity of interfaces.
Shape = Protocol
Color = Data type
An ESB performs the following
between requestor and service
• CONVERTING transport protocols between requestor
and service
• HANDLING business events from disparate sources
• TRANSFORMING message formats between requestor
and service
• ROUTING messages between services
WebSphere Software
6
WebSphere ESB and WebSphere Message Broker
ESB:
WebSphere ESB
Advanced ESB:
WebSphere Message Broker
Customers face a range of ESB requirements. As a result, any given
project might require an ESB or an Advanced ESB… OR BOTH.
Web Services connectivityand data transformation
Web Services
JMS
WebSphere MQ
HTTP
XML
WebSphere Adapters
JMS
TIBCO Rendezvous®
Web Services
Weblogic JMS®
WebSphere MQ
TIBCO EMS JMS®
HTTP
SonicMQ JMS®
Biztalk®
MQe Tuxedo®
Real-time IP
FTPMulticast
MQTT
XML
COBOLCopybook
EDI-X.12
ACORD
EDI-FACT
ebXML
AL3
HIPAA HL7
SWIFT FIX Custom Formats
Word/Excel/PDF
Plus the following:
WebSphereAdapters
Universal connectivity and data transformation
WebSphere Software
7
WebSphere Message Broker Overview
Message comesIn from
application/service client
ParseTo tree structureRoute + Transform
Parse
Message reachesDestination(s)
WebSphere Software
8
Message Formats in WebSphere Message Broker
� XML (with or without namespace support)
� Fixed-length Formats (COBOL, C, …)
� Tag Delimited Formats (CSV…)
� SAP iDOCs
� Binary
� EDI and industry standards
�HIPAA
�EDIFACT
�HL7
�ACORD
�SWIFT
�EDI-X12
�…
WebSphere Software
9
Message Broker Techniques For Transforming Messages
� Scripting via ESQL
� Scripting via Java
� Visual Drag & Drop Mapping
� Invocation of a Java Method
� XSLT for XML to XML
� WTX node (see later)
� Transactional database interactions
WebSphere Software
10
Message Sets
- Tell the broker how to parse a bitstring to an internal tree structure and vice versa
- Decouple the technical format from the business meaning of the data
- Allow parsing of all kinds of formats, far beyond XML
WebSphere Software
11
Message Sets (Ctd)
WebSphere Software
12
Creating the flows…
WebSphere Software
13
Message Flow overview: some I/O nodes
WebSphere Software
14
Some Non-Functional characteristics� Transactionality
�XA compliant resource coördinator
�z/OS: distributed transactions including message flow and a CICS transaction
� Performance & Scaleability�WebSphere Message Broker is designed to deliver performance…
� Highly performing processes
� possibility to create multiple Execution Groups (Processes) per broker
� possibility to process flows in Multiple parallel threads
� possibility to place multiple brokers in parallel
�Automatically scales to the # of CPUs available
� DRP & HASeveral scenario’s are possible:
� Parallel Sysplex (z/OS)
� HACMP
� MQ Clustering / Http loadbalancing
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
WebSphere TX Product Overview
WebSphere Software
16
A little history on Transformation Extender…
� 1983, TSI International
� Industry’s strongest transformation engine
� Strong MQSeries Partner
� Ascential acquired Mercator in 2003
� Renamed to DataStage TX
� IBM Acquired Ascential (2005)
� Renamed to WebSphere Transformation Extender
� Universal advanced transformation engine throughout the WebSphere portfolio
1983
2003
2005-2006
WebSphere Software
17
WebSphere TX
Input Card(s) Output Card(s)
MAP
TRANSFORM
Adapters
WebSphere Transformation Extender: A Powerful and Portable Transformation Environment
Native O/S
Embedded On IBM & Others
Executeson:
Through Enterprise Service
Bus
WebSphere Message Broker
WebSphere TXLauncher
Extending Process Server
Extending ESBWebSphere ESB
WebSphere ProcessServer
Available Now
Available Now
Available Now
WebSphere Software
18
WebSphere Transformation Extender is a strategic solution for our customers
� On average, over US$ 800B* in transactions flows through WebSphere TX on a daily basis�4 Major Stock exchanges
�9 of 10 Major US Financial Firms
�75% of Blue Cross / Blue Shield claims
�6 of 10 Major Insurance Firms
�3 of 3 Top US Pharmacy Benefits Providers
�6 of 6 Top Global Banks
�6 of 10 Top US Banks, 4 of 5 UK Top UK Banks
�Major Global and National Telecoms
�Over 95% of all US Department of Defense logistic transactions
�and on and on and on….
* $800B is based on available, tangible measurements. Actual estimates are in excess of $2.7T /day
WebSphere Software
19
MANUFACTURING, RETAIL & DISTRIBUTION
FINANCIAL SERVICESHEALTHCARE /
INSURANCE
WebSphere TX has significant Vertical Marketshare
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
Developing A Solution Using WebSphere TX
WebSphere Software
21
RuntimeMap
Designer
� Map Data Structures
� Defines Transformations
� Attach Adapters
Input Card(s) Output Card(s)
MAP
TRANSFORM
Type Designer
� Define Data
Structure
R
Tree
Importer
Database
Interface
Designer
Tree
Maker
Predefined
Trees
Application
Templates
Map
Source
Compiled
Map
The IBM WTX Integration Environment
DEPLOY
� Standalone
� Embedded
� WebSphere MB
WebSphere Software
22
Developing a WTX Solution
� We usually talk about three simple steps:
�Describe the data...
� Data that is handled by IBM WebSphere TX, are described in "Type Trees”
that define the data structure and semantics.
�Transform the data...
� Data is mapped between the source and destination using drag-and-drop.
�Deploy the transformation
Multiple deployment models:
Batch
Standalone Event Driven TX (installed base)
Tightly-coupled (MB Plug-in, Java API)
Loosely-coupled (RMI)
WebSphere Software
23
Describe the Data...
� IBM WebSphere TX’s Type Trees� The Type Trees defines the structure of the data to be handled in IBM WebSphere TX.
� This XML structure is then translated into an IBM WebSphere TX Type Tree.
WebSphere Software
24
Integration: Metadata� Automatic data definitions: Importers
� XML DTDs, Schema
� COBOL Copybooks
� Database Structures
� Application Definitions
� SAP: BAPI, Idoc, DXOB, BDC
� PeopleSoft 7: Business Document, Message Agent
� PeopleSoft 8: Application Messaging, Component
Interface
� Siebel: COM Business Object, EAI, EIM
� WSDL (Web Services)
� EJB API
� Java Class
� Java Messaging Service
� Other: Tuxedo, text files
• Built-in data definitions
– EDI, HL7, SWIFT,
ACORD, …
– All definitions are
modifiable through GUI
WebSphere Software
25
WTX Industry Accelerator Packs
WTX (any of the server types)
WTX Industry PACK’s
EDI
EDIFACT
TRADACOMS
ODETTE
SWIFT Net FIN
SWIFT Net Funds
*SEPA
*ACORD
*NACHA
HIPAA
HL7
NCPDP
* Available today as an ISSW offering
WebSphere Software
26
WTX (any of the server types)
WTX Enterprise Application PACK’s
SAP R/3
SAP XI
SiebelPeopleSoft
WTX Enterprise Application Accelerator Packs
Includes BAPI, ALE and IDOC integration capabilitiesRuns inside SAP App Server
Includes BAPI, ALE and IDOC integration capabilities
WebSphere Software
27
Built-in Validation (Restrictions)
Items have extensive validation options
• Enumerations• Exclusions (with substitution)
• Patterns• Ranges
WebSphere Software
28
Additional Built-in Validation
� Component Rules allow cross-field validation
� Incorporates semantic validation directly into the parsing process
WebSphere Software
29
Additional Built-in Validation
� Component Rules support aggregate functions
WebSphere Software
30
Transform the Data...
� The Type Trees for input and output are added to IBM WebSphere TX
Map Designer.
� For each input and output, or “card”, an adapter and a data source are defined.
� IBM WebSphere TX can handle several inputs to one or more outputs, and vice-versa.
� The mapping is a simple drag and drop from the input source to the output target.
� For each field in the output card, rules can be added--for example, mathematical formulas, conversions, etc. More than 150 different rules, similar to MS Excel formulas, are available through a self-describing wizard.
WebSphere Software
31
Sample mapping in IBM WebSphere TX’s Map Designer � A sample mapping in Map Designer:
WebSphere Software
32
• Drag and Drop
• No Procedural Coding!!
• Easily represent complex business rules
Enter transformation rules
WebSphere Software
33
Deploying and running the maps
� IBM WebSphere Transformation Extender
The core transformation engine is embedded within C, Cobol, Java applications though
the APIs which come with the product.
� IBM WebSphere Transformation Extender with Command Server
Suitable for scheduling and executing batch file transformation from command line, Unix
script and JCL Integration
� IBM WebSphere Transformation Extender for Message Broker
TX maps can be embedded in WebSphere Message Broker flows.
� IBM WebSphere Transformation Extender for ... (ESB, Process Server, Partner
Gateway)
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
WebSphere Transformation Extender For Message Broker
WebSphere Software
35
WMB Plug-in components
� Parser
�Transforms bit streams to Message Trees and Message Trees to bit streams
�WebSphere TX map can be used as a plugin parser in WebSphere Message Broker
�Used in MQInput node (Bitstream to tree structure) and MQOutput node (tree structure to bitstream)
�New Message Domain (DSTX/DTX)
� Plug-in node
�Transforms anything to anything
�Broker sends BLOB to DSTX and gets BLOB with map result back
� Icon installs into Message Flow palette
�Easy configuration
�Can make use of TX adapters
WebSphere Software
36
�Can be used to create a tree structure out of complex data structures.
� Tree structure can be accessed by ESQL or Java code
�Alternative for Custom written plug-in parsers
� TX map instead of coding c!
�Uses TX map which converts Complex data to XML (used by MQInput node
�Uses TX map which converts XML to Complex data (used by MQOutputnode)
WebSphere TX as a parser for WebSphere Message Broker
WebSphere Software
37
WebSphere TX as a parser for WebSphere Message Broker
Call TX map for parsing
WebSphere Software
38
WebSphere TX Plugin parser configuration
<MessageTypes>
<Type Name=“ZORDRS_01"
InputMapPath=“edi_xml.mmc" InputMapCard="1" OutputMapPath="xml_edi.mmc" OutputMapCard="1" />
</MessageTypes>
Message Type
� Create a Type Element in dstxwmqi.ini where the Name matches the Message Type (broker \bin directory)
� For MQInput nodes, InputMapPath attribute points to WebSphere TX map that transforms a bit stream to a Message Tree
� For MQOutput nodes, use OutputMapPath to point to WebSphere TXmap that transforms a Message Tree to a bit stream
DsTXmaps
WebSphere Software
39
Message Broker MQInput node to enable DsTXparser
Message Type
Plug-in name� Open properties of MQInput node
� Type DSTX intoMessage Domain
(not in dropdown yet)
� Message Type references type name in dstxwmqi.ini
WebSphere Software
40
WMB Parser Use Cases
� Convert any input stream to Message Broker logical tree
� Reuse existing ESQL
� Mix and match DSTX and standard parsers
� No need for custom written (c coded) Parsers
� Content Based Routing of very complex data structures
� TX map can be used as a plugin parser at the input node. It converts the complex data structure to the internal Message Broker logical tree.
�Message Broker performs the Content Based Routing
� TX map parses the logical tree back to Complex Data Structures
� This is just the beginning....
WebSphere Software
41
�Can be used to call any TX map from within a Message Broker flow
�Transaction Control of message broker is maintained
�Message Broker passes BLOB to TX map
�Plugin can pass BLOB back to message broker, or can parse to any other format: XML, MRM, ...
TX plugin node for WebSphere Message Broker
Call TX map for parsing
WebSphere Software
42
� Plugin node Installs into standard message flow pallete
�Plugin node can be used in message flow like any other node
TX plugin node for WebSphere Message Broker
WebSphere Software
43
WebSphere TX Plug-in Node Configuration
DSTX Map Name
� Open properties of Plug-in node
� Enter Map name and OutputCard
�Optionally, enter Output Domain, Set, Type, Format for MRM parsing�If these are empty, a BLOB is propagated
WebSphere Software
44
WMB Plug-in Node Use Cases
� Mapping between complex message formats� Message Broker native model (MRM) supports C, COBOL, XML and many tagged or string-delimited formats
� WTX adds support for custom formats that are hard to support with the MRM
� WTX EDI libraries are more comprehensive
� Rich Industry Support via WebSphere TX is fully leveragable in Message Broker
� Complex and Proprietary Data Types handled natively in TX
� Support for advanced transformation requirements� Multiple data sources with interdependencies
� Very efficient processing of large data records or messages
� Advanced data validation without complex coding
� Leveraging Message Broker’s Clustering capabilities for scalability� Deployment to multiple execution groups, brokers, machines
� Use of the TX adaptersTX offers it’s own set of adapters. Some of these adapters offer functionality which is not offered by any of the WBI adapters: (ZIP, …)
� File handling of very large files (up to 4 GB)TX is very fast in translating large files. The location of a file to be translated can be passed through by Message Broker to TX. TX finds the file and performs the translation which
� WebService enablement of existing TX maps
®
IBM Software Group
© 2005 IBM Corporation
November, 2006
Deploying WebSphere TX In An SOA
WebSphere Software
46
WebSphere TX can leverage the webservices capabilities of WebSphere Message Broker
WebSphere TX: powerful
Any-to-any mapping
WebSphere Message
Broker: powerful SOA
capabilities
•WSDL generation from Message Set
•Out-of-the- box SOAP message sets
•SOAP/JMS support •SOAP/HTTP(s) support •Webservice mode•WebService Client mode•...
WebSphere Software
47
WebSphere Message Broker: powerful SOA
capabilities
•WSDL generation from Message set
• Out-of-the- box SOAP message sets• SOAP/JMS •SOAP/HTTP•Webservice mode •WebService Client mode•...
WebSphere TX can leverage the webservices capabilities of WebSphere Message Broker
WebSphere TX: powerful
Any-to-any mapping
WebSphere Software
48
How to enable a DsTX map for WebServices using WMB example for SOAP/http
HTTPRequestInitial Processing:
-SOAP de-enveloping
-parse as BLOB
DSTX map
Final Processing:
-Parse as XML
-SOAP Enveloping
HTTPResponse
Web
Services
Client
SOAP/http
request
SOAP/http
response
WebSphere Software
49
WebService Enablement of DsTX maps: Howto
1. Read input
2. De-envelope and Parse content as BLOB
3. Pass BLOB to DsTX plugin
4. Parse output of Plugin as required and envelope
5. Reply
WebSphere Software
50
How to enable a DsTX map for WebServices using WMB - CASE
� A DsTX map (Fixed Format to XML) needs to be webservice enabled
� The map takes in non-xml data � needs encapsulation in XML to fit in a SOAP message
� If the map ends OK, the resulting XML data need to be passed back in the SOAP response
� If the map fails, a SOAP Fault needs to be returned, containing the Map error code thrown by DsTX
� Transport required is SOAP/http
� the map is designed to receive CRLF as a delimiter, but clients can send LF and we need to cope with that
WebSphere Software
51
The flow layout
Web Services Client
Strip SOAP
Parse asBLOB
MAPenvelope
reply
Read
WebSphere Software
52
What makes WebSphere TX Different?
One Engine,One Engine,
Multiple Multiple
Deployment Deployment
OptionsOptions
Ontological DataOntological Data
Model forModel for
All Data TypesAll Data Types
High High --throughputthroughput
Execution of ComplexExecution of Complex
Transforms andTransforms and
EnhancementsEnhancements
InIn--Process Process
Data ValidationData Validation CodeCode--FreeFree
Design andDesign and
DeploymentDeployment
WebSphere TXWebSphere TX
WebSphere Software
53
What value does WebSphere TX bring to WebSphere Message Broker users?
� Support for additional, complex message formats�Message Broker native model (MRM) supports C, COBOL, XML and many tagged or string-delimited formats
�WebSphere TX adds support for custom formats that are hard to support with the MRM
�WebSphere TX EDI libraries are more comprehensive
�Rich Industry Support via WebSphere TX is fully leveragable in Message Broker
�Complex and Proprietary Data Types handled natively in TX
� Support for advanced transformation requirements�Multiple data sources with interdependencies
�Very efficient processing of large data records or messages
�Advanced data validation without complex coding
� Support for TX adapters
� Large Data Size Processing�Messages/Files up to 4GB in size per transform
WebSphere Software
54
What value does WebSphere Message Broker bring to WebSphere TX users?� Delivers a powerful event processing framework for the transformation engine
�Extensible and open “message flow” framework integrates transformation with routing, validation, correlation and other rich event/message processing
�Complex Event Processing capabilities allow maps to be triggered by configurable patterns of event sequences
� Exploits the Enterprise Service Bus for effective data delivery and web service enablement
�Integrated environment for all aspects of transactional data movement in the enterprise
�Provides a complete application integration solution
�Leverages robust file transfer and messaging capabilities
�Allows easy inter-operation with ESB-enabled adapters and services
� Provides a robust, scalable environment for transformation services
�Ensures high availability and failover
�Exploits WMB clustering capabilities
WebSphere Software
55
Q & A
WebSphere Software
56
Enroll for the free Proof-Of-Technology!
� When ?
�06-12-2006 to 08-12-2006 08:30-17:00
� Where?
� IBM Forum Brussels
� Content?
�TX Lectures and hands-on lab exercises
�Usage of TX with and without WebSphere Message Broker
� Where to enroll?
�http://www-5.ibm.com/be/events/en/agenda.html
or
�http://www-5.ibm.com/be/events/en/middleware_appdev.html#362