Effective administration of IBM Integration Bus - Sanjay Nagchowdhury

1

InterConnect2017

HHI-2151Effective Administration of IBM Integration Bus

Sanjay Nagchowdhury

IBM Integration [email protected]

2

Effective Administration…

Installation

Configuration

MonitoringDeployment

Migration

Security

3


Installation

Configuration

Deployment

Migration

Monitoring

Security

4

Installation

Radically Simplified Packaging and Installation– Full function, simple, single package install– Developer Operating Systems contain Toolkit and Server

• Total size approx. 1.3 GB– Server Operating Systems contain only server

Other changes– Full entitlement to MQ remains– MQ no longer packaged

• Default queue manager for IB node for backwards compatibility

Built-in Unit Test Environment– Developer tools have built-in unit test server

• Fixed name of TESTNODE_<userid>– Started and stopped with tools– Can still test / deploy to manually created local and remote servers

Single install package for server and toolkit.

Simple unzip on unix.

Local integration node and server automatically created

when you start toolkit.

tar –xzf…

New Tutorials Gallery.

5


Installation

Configuration

Deployment

Migration

Monitoring

Security

6

Configuration

TOOLKIT• Connect to local or remote

Integration Nodes using toolkit• Configure bar file properties.• Deploy bar files• Configure policy sets for web

services

CONSOLE• New command console• iib help, iib tools, etc• Remote administration commands

need to point at the integration node’s administration port instead of queue manager.

Integration API• Java interface for administering IIB

through Java code.

REST API• Administer IIB resources using

operations defined on a REST model.

WEB UI• Primary graphical means of

administering IIB.• Many new features in V10.

• Manage Integration Servers

• Define policy• Create configurable

service• View Statistics• Workload Management

7

Manage Integration Servers• Start, stop• View, Create, Rename, Delete

Manage deployed resources• Start, stop, • View, manage, delete

Start/stop statistics and accounting data

Start/stop resource statistics

Start/stop flow monitoring for publishing events, record and replay and business transaction monitoring

Configuration

NEW !

8

Loopback Request Node

• The LoopBack Request node provides a way for IIB message flow authors to invoke synchronous CRUD operations to external systems using LoopBack Connectors.

• LoopBack provides numerous connectors to access enterprise and other backend data systems such as:• NoSQL databases.• Relational databases.

Cloudant

NEW !

Windows, Linux x86

Integration ServerNode.js®

MongoDBConnector

PostgreSQLConnector

CloudantConnector

Non DatabaseConnectors

Community Connectors

LoopBack

Cloudant

Data SourcesIntegration Node

MongoDB

9

ConfigurationNEW ! New LoopBack Request node provides a

way for IIB message flows to invoke synchronous CRUD operations to external systems using LoopBack Connectors.

LoopBack is an open source Node.js framework that can be extended by downloading and installing connectors from a large open source catalog using the ‘npm’ tool.

LoopBack provides connectors for:• NoSQL databases e.g. MongoDB, IBM

Cloudant• Relational databases e.g. PostgreSQL,

MySQL• Other backend services e.g. REST,

SOAP

10

LoopBack Connectors

Contributed by LoopBack

Community

http://loopback.io/doc/




11

Configuration

MQSI_WORKPATH\connectors\loopback\datasources.json

MQSI_WORKPATH\node_modules1

2

NEW !

3 mqsisetdbparms –n loopback::<secid> -u uid –p pwd

npm install loopback-connector-mongodb --save

12


Installation

Configuration

Deployment

Migration

Monitoring

Security

13

Toolkit

BAR files contain configurable properties which

allow you to deploy your applications from your QA

system to Production.

Web UI

Message node properties are configurable.

Workload management properties are configurable.

14

Deployment When a REST API is deployed, the Swagger document for that REST API is automatically

made available over HTTP from the same server and port that the REST API is hosted in.

The deployed Swagger document is automatically updated to reflect the server, port,

and HTTP/HTTPS details for the deployed REST API. You do not have to update it with the

correct details before deployment.

The REST APIs can be pushed to API Connect where you can secure them, define Service Level Agreements and

examine visual analytics data.

NEW !

How many Integration Servers should I have?How many Additional Instances should I add?

Additional Instances

• Results in more processing threads• Low(er) memory requirement• Thread level separation• Can share data between threads• Scales across multiple servers

Integration Servers

• Results in a new process/address-space• Increased memory requirement• Multiple threads including management• Operational simplicity• Gives process level separation• Scales across multiple servers

Recommended Usage

• Check resource constraints on system• How much memory available?• How many CPUs?

• Start low (1 server, No additional instances)• Group applications in a single integration server • Assign heavy resource users to their own integration server• Increment integration servers and additional instances one at a time

• Keep checking memory and CPU on machine• Don’t assume configuration will work the same on different machines

• Different memory and number of CPUs 15

Deployment Considerations Deployment

How many copies of a message flow do I need ?

Each message flow is different…..Each will use certain different level of resources (CPU, memory, I/O) and have a particular performance profile

In deciding number of copies to run need to know1. Availability Requirements2. Target throughput (messages/second)

rate3. Target response time

Number of copies of each message flow needed will vary.So do not decide the same number of copies are sufficient in all cases.

Ultimately need to balance resource manageability &

availability.

16


Installation

Configuration

Deployment

Migration

Monitoring

Security

17

Monitoring

Workload management policy offers dynamic control over:

• Message rate limits.• Additional instances.• Commit count & commit interval.• Automatic restart in case of

unresponsive message flow.

View message flow and resource statistics in the Web UI. Use this to analyse bottlenecks in your flows.

18

IBM Cloud Product Insights is a Bluemix service that is part of IBM Connect to Cloud.

MonitoringNEW !

20

Different IBM products can be registered, so you can create a cross-product inventory and

view product usage metrics across your portfolio of IBM products in your solution.

Monitoring

If you click on Register a product, you can see which IBM products are available for

registration.

21

You will see all products that are available for registration.

If you click on any of these, you will see instructions on how to register

your product.

22

You can see what is the minimum supported version of IIB.

Links are provided to the product web page and knowledge center.

Simple instructions on how to activate the Bluemix reporting

feature for your Integration Node and Integration Servers.

Click this to copy the command and then fill

in the blanks!

apihost and apikey information is on the Service credentials tab.

Restart your Integration Node for the changes to take effect.

23

The Service Credentials section is pre-populated with an apikey which has been

created on your behalf

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

You use the apihost and apikey when activating the Bluemix reporting feature.

24

You can group your Integration Nodes and Integration Servers into

groups. For example: Dev, QA, Stage, Production

Usage data can be seen for a group of Integration Servers or an

individual Integration Server.

25

Choose between:CPU time, Number of active CPUsResident Set Size

Usage Details Advisor

View Integration Node details

View a list of recommended

services from the Bluemix catalog.

26

[sanjayn@sachin iib-10.0.0.8]$ mqsichangebluemixreporting LINUX_NODE -g -l active -r us-south -o [email protected] -a "Sanjay Nagchowdhury" -u [email protected] -p XXXXX -dRequest Bluemix logging tokens: https://logmet.ng.bluemix.net/login

Successful HTTPStatus: 200

Publishing Kibana dashboard

Successful HTTPStatus: 200

Response: Your Kibana 4 dashboards can be accessed at the following location: https://logmet.ng.bluemix.net/app/#/kibana4The Kibana 4 dashboards that have been uploaded include: IBM Integration Bus Dashboard

BIP8377I: Current dashboard has been loaded into Kibana for Bluemix logging space 'Sanjay Nagchowdhury'. Dashboard files are also prepared for manual import into Kibana from '/home/sanjayn/iibconfig/BluemixReporting/LINUX_NODE'.

BIP8071I: Successful command completion.

Reporting logging information to a Logmet service in IBM Bluemix and displaying it in a Kibana dashboard

Run mqsichangebluemixreporting to configure a connection to the Logmet service in Bluemix and send it logging information.

Go to this URL in a web browser to see logging infromation in a dashboard. Command can be repeated for multiple

Integration Nodes and Integration Servers to see logging information in a single dashboard.

NEW !

https://logmet.ng.bluemix.net/login

27

Example using Integration Nodes running on Linux and Windows.

Top 5 hosts that are generating log events

Top 5 integration servers that are

generating log events

Top 5 log events across the Integration Servers for a timeslot.

All events by severity.

Custom visual which shows Integration Servers that are stopping/starting/changing

28

This spike shows that a sudden increase in error messages occurred.

You can zoom in and see precise BIP messages that were generated in this

timeslot.

29

Chart after zooming in on the timeslot to check the errors.

Bottom table shows the logging messages that were recorded from the different

Integration Servers.

You can see errors that were generated.

You can see errors that were generated.

30


Installation

Configuration

Deployment

Migration

Monitoring

Security

31

Security

Admin security is optional, it is not enabled by default– Control user access to Integration Node resources– Enable through mqsichangebroker / mqsicreatebroker

Administration Security

Authentication Authorisation

“You are who you say you are!” “But what are you allowed to do ?”

Credentials checked by….

Web user account with a local password.Local user authenticated by OS

Integration Node LDAP ServerWeb user account with a password held in the LDAP Server

Permissions checked using…

Queue-based authorisation File-based authorisation

SYSTEM.BROKER.AUTHSYSTEM.BROKER.AUTH.EG

MQ INQ

MQ PUT

MQ SET

File Read

File Write

File Execute

33

SecuritySet mode to file-based

authorisation

Add web user account with a local password

Web UI now requires userid/password

User can view the Integration Server and its contents but

cannot start/stop/deploy

Set permission to view Integration Server, but not start/stop/deploy

34

Security

XXXXXX

Set the LDAP server that you want to use for authentication.

Configure Integration Node to connect to the LDAP Server.

Add a web user account for a user that is defined in the LDAP

Server.

Use a wildcard to add all users.

NEW !

35


Installation

Configuration

Deployment

Migration

Monitoring

Security

36

Supported Migration Paths

You can migrate to IBM Integration Bus Version 10.0 from the following previous versions:

• WebSphere® Message Broker Version 7.0.0.5

• WebSphere Message Broker Version 8.0

• IBM Integration Bus Version 9.0

Migration

Co-existence: IIB V10 can co-exist with IIB V9, WMB V8 and V7.

Windows : • Multiple instances of each of the versions (but different fixpacks) can be installed on the same system.

UNIX and z/OS : • Multiple instances of each of the versions and fixpacks can be installed on the same system.

Ensure correct mqsiprofile is sourced before starting the Integration nodes to pick up the correct versions.

Multiple versions of IIB Toolkit can be installed on the same machine • You can not use IIB V10 Toolkit to connect to a V9 broker or vice versa

39

MigrationMigration Options In-place migration:

• Migrates existing broker and its components immediately to V10

• Should be performed on the same system where broker exists

• Requires stopping the broker / integration node

• Requires running the command: mqsimigratecomponents • Provides –t option to roll back to previous state

Parallel migration:

• Provides ability to migrate on same or a different system Does not overlap with the existing broker, so no need to stop the broker

• Create new Integration node and deploy artifacts to it

• Requires the Integration node / server properties and Administration security to be reconfigured on the newly created components

1

2

3

4

Install V10

Stop Broker

mqsimigratecomponents

Start Integration Node

1

2

3

4

Install V10

Create new Integration Node

Deploy artefacts

Stop old Broker

Backup resources first!

40

Demo !

MongoDB

Switch Server

LINUX_NODE

WINDOWS_NODE

43


Installation

Configuration

Deployment

Migration

Monitoring

Security

44

IIB Sessions at Interconnect 2017Session Who Time

2110A What's New in IBM Integration Bus BT Monday 16:15 – 17:00

2141A IBM Integration Bus Futures and Strategy (Inner Circle only) BT Tuesday 11:30 – 12:15

2158A Technical Introduction to IBM Integration Bus GG Tuesday 13:30 – 14:15

2118A Developing Integrations for IBM Integration Bus on Cloud GG Tuesday 14:30 – 15:15

2144A IBM Integration Bus Customer Roundtable BT Tuesday 15:45 – 16:30

2121A Docker and IBM Integration Bus GG Wednesday 09:00 – 09:45

2151A Effective Administration of IBM Integration Bus SN Wednesday 10:15 – 11:00

2144B IBM Integration Bus Customer Roundtable BT Wednesday 16:15 – 17:00

2124A Operational and Business Monitoring with IBM Integration Bus SN Thursday 09:30 – 10:15

2111A IBM Integration Bus and REST APIs SN Thursday 10:30 – 11:15

2166 IBM Integration Bus Version 10 Hands-On Scheduled Lab GG+SN Monday 13:00 – 14:45

9402 IBM Integration Bus Version 10 Hands-On Open Lab None Any Open Lab Session

In case powerpoint isn’t your thing …

https://developer.ibm.com/integration Lots of Blog entries, regular updates and links to product demo

videos! All our recent enablement material is on youtube

IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs

Running IIB in Bluemix Container Service https://youtu.be/ybGOiPZO3sY

IIB and Kibana dashboards https://youtu.be/sCPrT2dHKSs

IIB and Hybrid Connect https://youtu.be/gWbxIooq3_g

IIB and LDAP https://youtu.be/HrqY9MyfzNs

IIB LoopBack Request node https://youtu.be/rUK_OQ5-Anw

Using IIB to integrate with MongoDB and Cloudant https://youtu.be/Is1pphngUlM

Using IIB for REST, Graphical Mapping & Salesforce: https://youtu.be/XIK6QvNSHdY

IIB, Kafka and Twilio SMS: https://youtu.be/7mCQ_cfGGtU

Using Kafka with IIB https://youtu.be/kYv0crxL86Y

Consuming REST APIs using the IIB REST Request node https://youtu.be/C_6gPlrCHZQ

Easy demo of an IIB App Connect node https://youtu.be/StwPbOiFKzk45

https://developer.ibm.com/integration

https://youtu.be/sCPrT2dHKSs

https://youtu.be/ybGOiPZO3sY

https://youtu.be/sCPrT2dHKSs

https://youtu.be/gWbxIooq3_g

https://youtu.be/HrqY9MyfzNs

https://youtu.be/rUK_OQ5-Anw

https://youtu.be/Is1pphngUlM

https://youtu.be/XIK6QvNSHdY

https://youtu.be/7mCQ_cfGGtU

https://youtu.be/kYv0crxL86Y

https://youtu.be/C_6gPlrCHZQ

https://youtu.be/StwPbOiFKzk

46

Notices and disclaimersCopyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented

as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

47

Notices and disclaimers continuedInformation concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®,Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml



48

InterConnect2017

49

Back up

50


Topology Planning

Installation

Configuration

SecurityDeployment

Migration

Monitoring

High Availability

51

• What does your SLA say?• Agree SLAs with the business

(% uptime)• Scheduled and unscheduled

outages

• Ensure you have the technology to meet the SLA

• Redundancy in applications, • Clustering, • PowerHA (HACMP),• What about applying service?

Availability % Downtime per year Downtime per month Downtime per week

90 (one nine) 36.5 days 72 hours 16.8 hours

99 3.65 days 7.2 hours 1.68 hours

99.9 8.76 hours 43.2 min 10.10 min

99.99 52.6 min 4.32 min 1.01 min

99.999 (“five nines”) 5.26 min 25.9 secs 6.05 secs

99.9999 31.5 secs 2.59 secs 0.605 secs

A CB1L1 L2

B2

A CBL1 L2 89.996%

Availability[~2.4 hours down per day]

98.996%Availability[~14.5 mins down per day]

Example• Same processing components: A, L1, B, L2, C

• Same levels of availability for each component• A, C, L1, L2 are 99.999% available. • B is 90% available

• So duplicate B B1 and B2• Individually B1 and B2 are 90% available, but

together they are 99% available.

• Increased availability by duplicating components

• How do I ensure that the integration node is continually processing messages?• Active/Active vs. Active/Passive

52

High Availability Each Integration

Node operates independently

Continuous availability of the service during a failure

Single instance of IIB

Active & Passive state must have identical copies of persisted data.

53

High Availability

Multi-instance queue manager• Integrated into the IIB and MQ products• Faster failover than HA cluster• Delay before queue manager restart is much shorter• Runtime performance of networked storage must be considered• IP address of standby instance is different to primary• No automatic fail-back to primary hardware when restored

HA cluster• Capable of handling a wider range of failures• Failover historically rather slow, but some HA clusters are improving• Some customers frustrated by unnecessary failovers• Extra product purchase and skills required

IP address of each machine is different.

Queue Manager restart quicker.

Single IP address.

Handle wider range of failures.

54

High Availability

Planning for disaster recovery• What would you do if your primary IIB location goes down?

• Distribute IIB to multiple sites if possible• This introduces data replication and latency concerns

• Keep DR concerns separate from HA!

Disaster Recovery High AvailabilitySystems at multiple sites with replicated configurations

Systems at a single site with a single configuration

DR is unplanned. An HA failover can be a planned activity.

Take regular backups!

Write a DR plan!

Test your DR plan!

55


Topology Planning

Installation

Configuration

SecurityDeployment

Migration

Monitoring

High Availability

QM1

IntegrationNode 1

QM2

IntegrationNode 2

sprayer

MQ requests

Machine A Machine B

Integration Node 1 has flows getting/putting to QM1 and QM2

Integration Node 2 also has flows getting/putting to QM1 and QM2

Topology Planning

QM1

IntegrationNode 1

QM2

IntegrationNode 2

sprayer

MQ requests

Machine A Machine B

If QM1 goes down, Integration Node 1 and Integration Node 2 are unaffected. They can continue to get/put from QM2

QM1

IntegrationNode 1

QM2

IntegrationNode 2

sprayer

MQ requests

Machine A Machine B

If Integration Node 1 goes down, but QM1 is still running, then Integration Node 2 is unaffected. It can continue to get/put from QM1 and QM2

Topology Planning

NEW !

Node wide HTTP listener

SYSTEM.BROKER.WS.INPUT

http://localhost:7080 https://localhost:7083

http://localhost:7080/ServiceC

http://localhost:7080/ServiceD

HTTPHTTPS

SYSTEM.BROKER.WS.REPLY

http://localhost:7080/ServiceA

http://localhost:7080/ServiceB

Integration Server

Integration Server

Integration Node wide HTTP(S) Listener

biphttplistener









Integration Server

Integration Server


Servlet Container

Proxy ServletHTTPHTTPS

biphttplistener

60










Threads

HTTP(S) Listener


Threads

HTTP(S) Listener

Integration Server

Integration Server



HTTPHTTPS

HTTPHTTPS

Integration Server

Integration Server


Integration Server HTTP(S) Listener

Servlet Container


biphttplistener

61










Threads

HTTP(S) Listener


Threads

HTTP(S) Listener

Integration Server

Integration Server



HTTPHTTPS

HTTPHTTPS

Integration Server

Integration Server


Integration Server HTTP(S) Listener

Servlet Container


biphttplistener

Export port configurations and use

in an external HTTP Server

Export port configurations and use

in an external HTTP Server

62


• Most resilient and best performing systems are those that are:• Loosely coupled and have parallel execution (threads and processes)• Implications for your message flow design will depend on the systems that IIB interacts with

• Vital to conduct performance testing before production in production like environment• Gives time to evaluate and refactor code if needed

• Top Tips• Always ensure your environment is reproducible• Treat DR and HA separate• Ensure regular backups• Encourage developers to create message flows that enables operational tweaks to be made• Schedule regular maintenance windows

Summary

Effective administration of IBM Integration Bus - Sanjay Nagchowdhury

Software

Transcript of Effective administration of IBM Integration Bus - Sanjay Nagchowdhury