Deploying and managing IBM MQ in the Cloud

Rob ParkerIBM MQ Development

1 7/10/2017

Deploying and Managing IBM MQ in the cloud

• Deploying and Orchestration

• Managing your Queue Manager In the cloud

• Monitoring your Queue Manager In the cloud

• Questions

Agenda

On-prem

… IBM MQ Appliance

AWSAWS

AWS

Cloud

Message Hub

IBM Bluemix(including Softlayer)

Distributed platforms

…

Private cloud

Run MQ, exactly how and where you need it

Deploying & Orchestration

4 7/10/2017


https://developer.ibm.com/messaging/mq-on-cloud/


Installing MQ

• Initial installation

• Applying fixpacks

• Migrating between versions

• Managing licenses

Running queue managers

• Create/start/stop/delete QM

• Naming queue managers

• Binding apps to QMs

• Failing over

Provisioning an Environment

• Create VM, start docker container etc.

• Setup virtual network interface

• Create storage

• Create security policies

Provisioning an environment

Manual Configuration and in-house management tools

Cross Cloud & Multi Resource management tools

9

Manage and script software installation

Similar to Chef, use in conjuction with an MQ install

package to automate and repeat installations

Provides an MQ cookbook to include in Chef recipes

Use to perform repeatable & automatable MQ

installations

Create immutable OS images with MQ pre-installed

Cookie-cutter installations of e.g. MQ & Redhat

Deploy images to chosen IaaS service

Installing MQ


10

Simple shell script can be a useful lowest

common denominator

Use in a heat template, docker image, VM…

Not dependent on specific framework

As well as being able to install MQ, can create

and start queue managers

Can combine with installation steps to give a

complete MQ environment

Creating & starting QMs


11

Deleting resources when they’re no longer needed

Orchestration isn’t limited to creating resources

Tear down resources when they’re no longer required

Quickly spin up everything necessary to run a scenario, tear everything down when it’s no longer

needed

Useful for running one-off tests, creating short-lived development environments

Keeps cloud costs to a minimum – only pay for what you need at the time

Don’t need to manually keep track of what you created

Repeatabilityterraform apply

terraform destroyhttps://developer.ibm.com/messaging/mq-on-cloud/

12

Using tools to create a full QM

If you use Packer to create an image

You can use scripts to provision, deploy that image and start/configure the QM


• MQ 8.0.0.4+ supported to run inside a Docker image

•Details: https://ibm.biz/mqdocker

• Brings the benefits of Docker to MQ

•Lightweight containers for running MQ

•Predictable and standardized units for deploying MQ

•Process, resource and dependency isolation

Binary image in Docker Hub Source in GitHub

Sample MQ Docker Container – Install and basic QM setup in 1 command

“docker run –e LICENSE=accept –e MQ_QMGR_NAME=QM1 ibmcom/mq”

Downloads MQ dockerimage from DockerHub

Starts MQ dockercontainer

Creates QM1

Starts QM1

Runs optional mqsc config script

1 2 3 4 5

MQ Advanced for Developersin the Bluemix Container Service

It is now possible to spin up MQ queue managers in the Bluemix Container Service with a couple of clicks

The fastest way to get up and running with MQ for development and experimentation

Pre-configured defaults mean instant access for administration and messaging applications

Managing your Queue Manager

15 7/10/2017

MQ

MQ

MQ

Auto-scaling

MQ

x n

• Key benefit of cloud technologies

• Change your infrastructure to meet demand

• Pay for what you’re using

• Cookie-cutter model using immutable software stacks

App

App

App

App

x n

AWS Auto-Scaling & Route53 DNS

EC2 auto-scaling group

MQ

EC2 Instance

MQ

EC2 Instance

MQ

EC2 Instance

MQ

EC2 Instance x n

• Manually or automatically scale the number of EC2 instances

• Scaling policies define when instances scale up or down

• Scheduled – e.g. scale up every

Monday morning, scale down Monday

evening

• Based on metrics - e.g. when queue

depths stay above a certain level for

15 minutes

• Have a listening app register new instances in DNS with unique hostname

AWS Service Discover Reference Architecture

https://aws.amazon.com/blogs/compute/service-discovery-an-amazon-ecs-reference-architecture/

AWS Auto-Scaling

EC2 auto-scaling group

MQ

EC2 Instance

MQ

EC2 Instance

x 1

• Auto-scaling not just for very dynamic systems

• Also appropriate for MQ failover scenarios

• Auto-scaling group size set to 1

• Monitoring process checks the health of the instance and replaces it if it has died

Networked file system or block storage

(e.g. AWS EFS)

Kubernetes – Create clusters of MQ Containers

StatefulSet

mq0 mq1 mq2

mq0

data

mq1

data

mq2

data

ReplicaSetGet

client

IP Service

Auto-Scaling Challenges

Challenges

• New queue managers need naming

• Permanently removing instances should only be done when queues are empty

• Application queues

• Transmit queues

• Dead letter queue

• If you’re using clustering, samples like amqsclm can help move your data to another queue manager before deleting it

• Clients need to be able to handle changing Queue Managers

MQ(?)

EC2 Instance

Important

Messages

App

?

MQ Infrastructure

Service Discovery

• Runtime components understanding where they need to connect to

• MQ clients need to know where they can find a queue manager

• MQ queue managers need to know where they can find a queue

• MQ partial repositories need to know where the full repositories are

Registration

• Administrators & planners need to know what they’re running and where

• Out dated versions of MQ

• Installations that need patching

• Queue managers that are underutilised

• Queue managers that are close to capacity

Auto-Scaling Challenges

Admin

App

?

Constraints to bear in mind

• QM-scoped artefacts

• Durable subscriptions only exist where they were created

• Likewise for global transactions

• Performance

• Putting to the same QM as a getter is preferable to multiple

QM hops

• Availability

• A queue manager with no getters may have orphaned

messages

HTTP CCDTs

App

App

App

App

MQ

MQ

MQ

MQ

HTTP

Server

MQ

/var/mqm/qmgrs/QMx/clntconn/CLNTCHL

runmqsc -c

HTTP GET

MQCONN(…)

MQ MQ

MQMQ

MQ MQ

MQMQ

MQ MQ

MQMQ

+

Centralised Registration

Bluemix

IBM Cloud Product Insights – New IBM Bluemix Service

• Supported in MQ 9.0.2+

• IBM Bluemix service that gives you an overview of your on-premise and cloud-hosted products

• See which queue managers (or Liberty/WAS servers, or IIB Brokers etc.) you have, which versions they are, what platforms they’re on, how utilised they are.

https://developer.ibm.com/product-insights

V 9.0.2

IBM Cloud Product Insights V 9.0.2

IBM Cloud Product Insights

Products listed in one place to drill down into

Individual queue managers displayed, showing their hostname, installation path, QM name and QM ID

V 9.0.2


Selecting a queue manager displays information about the queue manager

MQ version, install path, platform etc.

V 9.0.2


Selecting a queue manager also displays basic usage information

Usage information about the queue manager (no. of put messages & bytes)

V 9.0.2


• Queue manager authorises itself using the API key associated with the Product Insights service

• Copy API key to file in /var/mqm (or Windows equivalent)

• Update qm.ini with the “BluemixRegistration” stanza

V 9.0.2

Monitoring your Queue Manager

31 7/10/2017

Monitoring & Logging


MQ MQ

MQMQ

MQ MQ

MQMQ

MQ MQ

MQMQ

+

Centralised Logging

MQ MQ

MQMQ

MQ MQ

MQMQ

MQ MQ

MQMQ

Centralised Logging • Amazon Cloudwatch• Elasticsearch• Splunk• IBM Product Insights Log Management• …

IBM Cloud Product Insights Log Management – Experimental Bluemix Service

• Based on Elasticsearch and Kibana

• Pre-defined Kibanadashboards show basic overview of your MQ estate

• Customisable to show different perspectives on different dashboards

IBM Cloud Product Insights Log Management

• Help pane describes the dashboard and how to use it

• Raw AMQERR messages for detailed investigation

• AMQERR activity over time, per queue manager

• Interactive – select a queue manager in one widget and the entire dashboard updates

Centralised Monitoring – MQ $SYS topics

• First introduced on the MQ Appliance to aid remote monitoring & diagnosis

• Added to MQ distributed platforms in MQ V9

• Benefits over pre-V9 statistics messages

• They automatically get published when an app subscribes

• Metadata topics let you see what’s available to subscribe to

• e.g. $SYS/MQ/INFO/QMGR/MYQM/Monitor/METADATA/CLASSES

• New data available that wasn’t previously (e.g. DISK and CPU activity)

• Includes MQ activity trace capability

• e.g. $SYS/MQ/INFO/QMGR/MYQM/ActivityTrace/ChannelName/MY.SVRCONN


• Separated from the rest of the topic tree

• Root of the $SYS/MQ tree is

SYSTEM.ADMIN.TOPIC object

• No permissions inherited from objects

above SYSTEM.ADMIN.TOPIC

• Wildcard subscription at “#” doesn’t match

anything under $SYS/MQ/…

• Wildcard subscriptions under

$SYS/MQ/… are restricted

MQ MQ

MQMQ

MQ MQ

MQMQ

MQ MQ

MQMQ

Centralised Monitoring

MQ app consuming

$SYS/MQ messages

Managed subscriptions to “$SYS/MQ/…” with remote queue as destination

MQ


• Visualise queue depths, CPU & disk spikes, # of client connections….

• See https://github.com/ibm-messaging/mq-golang to try it for yourself

https://github.com/ibm-messaging/mq-golang

41 7/10/2017

Prove it

“I can put and then get a message from a new Docker container running MQ in a clean workspace in about 5 mins… 1.5 mins with the image already downloaded.”

--Robert Parker, June 2017

Questions?

42 7/10/2017

43 7/10/2017

Notices and disclaimers

Copyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.

U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.

Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.

IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”

Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.

Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and

the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.

References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.

Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.

It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actionsthe customer may need to take to comply with such laws. IBM does not

provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.

44 7/10/2017

Notices and disclaimers continued

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.

The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.

IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®,Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.

http://www.ibm.com/legal/copytrade.shtml

Deploying and managing IBM MQ in the Cloud

Software

Transcript of Deploying and managing IBM MQ in the Cloud