Mike Willbanks | Barnes & Noble

Message Queues : A Primer

Housekeeping… •  Talk

– Slides will be posted after the talk. •  Me

– Sr. Web Architect Manager at NOOK Developer

– Prior MNPHP Organizer – Open Source Contributor – Where you can find me:

•  Twitter: mwillbanks G+: Mike Willbanks •  IRC (freenode): mwillbanks Blog:

http://blog.digitalstruct.com •  GitHub: https://github.com/mwillbanks

Agenda •  Message Queues? •  Protocols and Software •  Picking a Message Queue •  Best Practices

A Definition “Message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. They use a queue for messaging – the passing of control or of content.”

What is messaging? “Messaging describes the sending and receiving of data (in the form of messages) between systems. Messages are exchanged between programs or applications, similar to the way people communicate by email but with guarantees on delivery, speed, security and the absence of spam.”

General Anatomy

Task

Producer ConsumerMessages

Messages

Messages

Producer creates a message and pushes it to the queue; the consumer reads from the queue and processes the message.

l  Pub/Sub l  FIFO buffer l  Push / Pull l  A way to communicate between

applications / systems. l  A way to decouple components. l  A way to offload work.

Describing Message Queues

l  Offload Heavy Work l  Integration with Legacy Systems l  Asynchronous Processing l  Parallel Processing l  Process consistency l  Scalability

Why to use a Message Queue

Unix Foundations l  “Write programs that work together.” l  “Do it in the background.”

l  Web systems need to be geared to run things asynchronously.

l  Distribution of load l  System integrity

Why it matters

MESSAGE QUEUE EXAMPLES

You’ve seen them before; they are used in most applications to help them scale.

GENERIC USE CASES When to make use of message queues in case you’re wondering.

Notifications Email, SMS, Push Messaging….

Photo Processing Thumbnails, Resizing, Watermarking, Converting…

Video Processing Resampling, Audio Overlay, Type Conversion…

Analytics Web Server Logs, Log Aggregation, PHP Errors, etc.

Integrations Save local first; push second.

PROTOCOLS

AMQP STOMP XMPP Vendor Specific

AMQP Advanced Message Queuing Protocol

l  AMQP Working Group (Community and Vendor)

l  Platform agnostic protocol. l  Completely open, interoperable and broadly

applicable. l  Many severs available and many client

libraries.

Overview of AMQP

How it Works Producer, Exchange, Queue, Consumer

l  AMQP utilizes exchanges, queues and bindings.

l  An exchange are routers with routing tables. l  A binding defines the routing rules. l  A queue is where the messages wait for a

consumer.

How it Works

Exchanges

l  Fanout Exchange l  No routing keys involved. Any message that is

sent to the exchange is sent to all queues bound to that exchange.

l  Direct Exchange l  Routing keys involved. A queue binds to the

exchange to request messages that match a routing key exactly.

l  Topic Exchange l  Routing keys involved. A queue binds to the

exchange to request messages that match a routing key pattern.

Understanding Exchanges

Implementations www.rabbitmq.com Very popular and common message queue owned by VMware.

qpid.apache.org Long standing project; apache foundation.

www.openamq.org Long standing project; ZeroMQ partner, no news since 2009.

l  An exchange, queue and bindings must be defined first. Publishing can then commence after. l  Create the queue l  Create the exchange l  Bind to the queue.

Building a Queue

l  Default behavior is no persistence. l  How important are the messages? l  Just about all items have a level of persistence if you

would like them to survive on reboot. l  Mark exchanges, queues and messages as DURABLE.

Persistence?

l  Extension compatible with AMQP specification 0-9-1.

l  pecl install amqp

PECL AMQP

AMQP Client

AMQP Worker

STOMP Simple (or Streaming) Text Orientated Messaging Protocol

Overview l  Simple protocol

l  Behaviors follow very simple commands.

l  Most message queues can communicate over STOMP.

How It Works l  When you send in a message, you tell it which

queue to go to. l  When you subscribe you request a queue.

Connect Send Disconnect

/queue/msg

PHP

STOMP

SERVER

Connect Subscribe Disconnect

/queue/msg

Read

Ack

Sever Implementations activemq.apache.org One of the oldest message queues existing; a apache foundation project

activemq.apache.org/apollo Next generation ActiveMQ

www.rabbitmq.com Very popular and common message queue owned by VMware.

www.jboss.org/hornetq Supported by Red Hat Middle Ware division, picking up steam.

l  pecl install stomp l  That was easy J

PECL Stomp

STOMP Client

STOMP Worker

XMPP Extensible Messaging and Presence Protocol (Although not really a “Message Queue”)

Overview l  Best for real-time data. l  Leveraging pub/sub can turn it into more of a generic message

system. l  Multiple libraries

l  JAXL - https://github.com/abhinavsingh/JAXL l  Xmpp - https://github.com/alexmace/Xmpp

XEP-0060: Publish-Subscribe l  Specification for implementing Publish Subscribe

models. l  Extension to the original XMPP specification.

Publish

Subscribe

Sub1 Sub2 Sub3

to, id, message

from, to, id, message

Publish

Message

SPECIAL MESSAGE QUEUES Various others; including Gearman, ZeroMQ, etc.

l  There are job servers available that are more flexible or more specific. l  Extreme Flexibility l  Job Severs l  Cloud Messaging

Overview

ZeroMQ The ultimate in message queue flexibility. Socket library that acts as a concurrency framework. Contains a PHP extension.

Several Types of Queues Request / Reply Publish / Subscribe Parallel Pipeline Fair Queuing And more…

ZeroMQ Client Example

ZeroMQ Worker Example

l  Application framework for farming out work. l  Job sever for asynchronous or synchronous

messages.

Gearman

l  Pass a job to the job server l  Worker receives the job and processes l  Ability to persist if enabled; default is in-memory.

Gearman

Gearman Client Example

Gearman Worker Example

l  Asynchronous Job Queue l  Good scheduling system (aka delays). l  User land PHP clients.

Beanstalkd

l  Pass a job to the job server l  Worker receives the job and processes l  Ability to persist to binlog; default is in-memory

Beanstalkd

Application Code

Beanstalkd Client API(Pheanstalk)

Beanstalkd Server

Beanstalkd Worker API(Pheanstalk)

Worker Application Code

Pheanstalk Client

Pheanstalk Worker

PICKING A MESSAGE QUEUE What you want to look for.

The Trifecta

Scale

Performance Durability

Messages

Horizontal?Vertical?

In-Memory?

Persistence?

Replication?Raw Speed?

# Messages?

Messages Messages

Messages

MessagesMessages

MessagesMessagesMessages

Standards •  A recognized standard?

– AMQP? STOMP? XMPP? •  How many developers?

– Will an unfortunate event kill off the product?

•  Undocumented protocol? – Forget about it!

Delivery Policies

Store Message Saved?

Process Message ACK?

Receive Message TTL?

Message Policies •  Handling in-active messages? •  Messages that failed processing? •  Time to live? •  Retry? •  Ability to check message status?

Routing Policies •  Fanout? •  Direct? •  Topic? •  Broadcast? •  Etc?

Security Policies •  Require authentication?

– LDAP/AD integration? •  Connection Restrictions? •  Require logging?

– Compliance –  Intrusion Detection

BEST PRACTICES Rules of the road my friends…

Messages •  Formatting

– JSON or XML are great options. •  Please no serialized PHP objects.

•  Message Size – Only as large as necessary.

•  Don’t send a binary object through the queue.

Workers •  Dumb as possible!

– Single message type •  Easier to scale.

– Single operation •  Easy to debug and far more flexible.

PHP Daemons •  Prevent Memory Leaks

– Detection – Cycle Workers

•  Handle Signals – Properly shutdown! – Watch out for OS service kills

•  Sleeping is good J

Environment Integration

Web Server

Database

Worker Server

Message Queue

Web Server -> Message Queue Worker Server -> Message Queue

High Availability

Web Server

Database

Worker Server

Message Queue

Message Queue

Worker Server

Send to 1

Insert multiple message queue servers + multiple worker nodes. Each worker node can connect to as many message queue servers as necessary.

Supervisord Process monitoring, logging and more!

Installing Supervisord •  Requires: python-setuptools

sudo easy_install supervisor!

•  Generate Configuration sudo echo_supervisord_conf > /etc/

supervisord.conf!!

QUESTIONS?

These slides will be posted to SlideShare & SpeakerDeck. SpeakerDeck: http://speakerdeck.com/u/mwillbanks Slideshare: http://www.slideshare.net/mwillbanks Twitter: mwillbanks G+: Mike Willbanks IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com GitHub: https://github.com/mwillbanks