IBM Message Hub service in Bluemix - Apache Kafka in a public cloud

1 © 2016 IBM Corporation

IBM Message Hub service in BluemixApache Kafka in a public cloud

Andrew SchofieldIBM Hursley Park


A scalable, distributed, high throughput message bus for your cloud-native applications

Connects services inside Bluemix with your systems beyond

Connects microservices, using open standard protocols

Streams data to analytics services for real-time insights

Built on Apache

What is Message Hub?


Bluemix is an open-standards, cloud-based platform for building, running, and managing applications

Build your apps, your wayUse the most prominent compute technologies to power your app: Cloud Foundry, Docker, OpenStack.

Extend apps with servicesA catalog of IBM, third party, and open source services allow the developer to stitch an application together quickly.

Scale more than just instancesDevelopment, monitoring, deployment, and logging tools allow the developer to run and manage the entire application.

Layered securityIBM secures the platform and infrastructure and provides you with the tools to secure your apps.

Deploy and manage hybrid apps seamlesslyGet a seamless dev and management experience across a number of hybrid implementations options.

Flexible pricing Try compute options and services for free and, when you’re ready, pay only for what you use. Pay as you go and subscription models offer choice and flexibility.

What is IBM Bluemix?


Start with a boilerplate


Or a runtime for building your app from scratch


Choose from a wide range of services to use in your app


Including Message Hub


Message Hub is Kafka in Bluemix


Bind to the service to get your credentials


Why are we using Kafka?Kafka has many valuable characteristics

• Fast• High throughput to handle whatever you throw at it

• Scalable• Easily add more brokers to increase throughput

• Durable• Consumers can be stopped for maintenance and catch

up without impacting the service

• Distributed• Replication and failure characteristics make managing

the service easy


Kafka in Bluemix

• Some services in Bluemix integrate well with Kafka’s streaming data-flow model

• For example, analytics such as Spark• Exposing, not hiding, the Kafka interface gives an event-driven

programming interface for application code

• Some internal services also use Kafka as plumbing

• Some compound services make use of Kafka for internal communication

• Think SaaS built using a PaaS

• Message Hub is intended to support all of these scenarios


Why might you use Kafka on the cloud?

• The benefits of Kafka are applicable to services in the cloud

• For example, feeding data into real-time analytics

• Serious use of any software takes investment• Either grow your own skills• Or pay someone else to support you

• Pay-as-you-go pricing makes it a cost-effective platform for experimentation

• For example, try some analytics on telemetry data from your mobile app


Kafka use cases which fit with Message Hub nicely• Messaging

• Replacement for a traditional message broker

• Website activity tracking• High-volume feeds of site activity such as views,

searches

• Metrics• Aggregating operational monitoring data

• Log aggregation• Copying physical log files from servers to central point

• Stream processing• Data flowing on the topics between processing stages


Kafka use cases which don’t fit so well• Event sourcing

• State changes are logged as time-ordered sequence, rather than storing the current state

• Commit log• Assists with resynchronisation of failed nodes

• Change data capture• Take a feed of database changes and store them as a

compact change history in Kafka, and feed data into multiple consumers from Kafka

The difference is that log data needs to be retained and perhaps compacted


In the public cloud, you’re sharing resources• Two instances of Message Hub available now

• Dallas and London

• Each is a 5-node cluster, shared among the tenants• Cluster-wide configuration is common

• Replication factor: 3• Clean-up policy: delete• Log retention: 24 hours• Maximum message size: 1 MB

• If you need more performance or isolation• Bluemix Dedicated

• Dedicated hardware in IBM datacenters, IBM ops team• Bluemix Local

• Customer hardware, IBM ops team


Pricing


Pricing example

• A workload of 10 msg/second published to a topic with 1 partition

• 26,784,000 messages in a month• About $25


Modifications to Kafka for use in the cloud• Client connections must provide credentials

• Username or API key tell us which tenant is connecting

• Client connections are always encrypted• Connections are over the public internet

• Real topic names have a tenant-specific prefix• Kafka topics are not hierarchical

• Number of partitions is capped per tenant

• Per-tenant metering data used to generate bills


Topic management• Message Hub topics do not auto-create

• Auto-create can only give a default topic config anyway

• Kafka users can use kafka-create-topic.sh• You can’t access zookeeper or a shell

• Two options for Message Hub:• Admin REST API for create/delete/list topics• Bluemix UI


Topic management in the UI


Topic configurationProperty Can set in MH? Default Description

cleanup.policy No “delete” “compact” would enable log compaction use cases, but would consume more storage

delete.retention.ms No 24 hours Applies to log compaction

retention.bytes No 1 GB Maximum log size

retention.ms Not yet 24 hours How long a log segment is retained

partitions Yes 1

replicas No 3

Defaults are taken from the cluster config, and that’s common for all tenants


Message Hub deployment architectureRed zone (internet)

Yellow zone (DMZ)

Green zone (internal)

FIREWALL

FIREWALL

App using Kafka client

HA proxyHA proxy

Kafka ZK

Bare-metal

Bluemix services: Logging, billing, authentication

Metering

Alerting

Cloudant DB

TLS

TLS

Bare-metal


Continuous delivery and microservices

• Message Hub is developed using a continuous delivery process and a microservice architecture

• It’s built out of small pieces that we can enhance rapidly

• Multiple deployment environments with promotion from dev/test towards production

• We can deploy changes into production as often as required, even several times as day if we had to

• When there’s a critical patch for Kafka, we…• Build the patch as deploy to dev/test• Test it thoroughly• Evaluate suitability for promotion to staging…• And repeat until the patch reaches production


Example – multiple partitions• The first delivery of Message Hub actually only

supports one partition per topic• Deliver quickly and iterate often• We are about to add multiple partitions

• Poised in our staging environment awaiting promotion• Includes:

• Admin API• UI• Metering


Performance• Throughput of the cluster is about 300,000 msg/s

• 100-byte messages• Secure connection from public network• Disk encryption of Kafka log• Secure connections for all user data, even within DMZ

• Latency ranges from 20ms to 100ms, average 50ms


Involvement in the community• Contributed several patches, particularly as release

of 0.9 approached• Primarily related to SSL/TLS and SASL

• Planning to propose Kafka Improvement Proposals shortly to augment SASL capabilities

• Have submitted sessions for the Kafka Summit in April 2016


A scalable, distributed, high throughput message bus for your cloud-native applications

Connects services inside Bluemix with your systems beyond

Connects microservices, using open standard protocols

Streams data to analytics services for real-time insights

Built on Apache

What is Message Hub?


Questions?

Twitter: @IBMmessaging

DeveloperWorks:https://developer.ibm.com/messaging/message-hub/

Bluemix blogs:https://developer.ibm.com/bluemix/blog/

LinkedIn: ibm.biz/ibmmessaging

Message Hub service:https://console.ng.bluemix.net/catalog/services/message-hub

IBM Message Hub service in Bluemix - Apache Kafka in a public cloud

Technology

Transcript of IBM Message Hub service in Bluemix - Apache Kafka in a public cloud