DO YOU QUEUE?ZendCon 2015
I AM MIKE WILLBANKS
• Father, Husband, Developer
• VP of Development at Packet Power
• Twitter : @mwillbanks
Task
Producer ConsumerMessages
Messages
Messages
WHAT IS A QUEUE?• Pub/Sub
• FIFO buffer
• Push / Pull
• A way to communicate between applications / systems.
• A way to decouple components.
• A way to offload work.
WHY QUEUE?
THE LONG RUNNING CASE
Request Long-process
Send ResponseResponse
THE INTEROP CASE
Request Conditioning
Service ResponseResponse
Web Service Call
Send Response
NOT YOUR DATABASE
Request Process
Response Send Response
DB
SO WHY QUEUE
• User experience
• System Security
• Load Distribution
• System Reliability
IN PRACTICEYou’ve seen this before…
WHAT YOU MIGHT QUEUE
COMMUNICATIONS
• Emails
• SMS
• Push Notifications
IMAGES
• Conversions
• Resize
• Thumbnail
• Watermark
VIDEO
• Conversion
• Resampling
• Audio overlay
IOT
• Receive messages from devices and process responses
PATTERNS
POINT TO POINT
Point to Point Channel
Receiver
Sender
PUBLISH / SUBSCRIBE
Publiser Subscribe Channel
Subscriber
Subscriber
Subscriber
MESSAGE BUSApplication
ApplicationApplication
Message Bus
PIPELINESender
Receiver
Point to Point Channel
Receiver
Point to Point Channel
INVALID MESSAGE
Channel
Receiver
Sender
X
Invalid Message Channel
PROTOCOLSOr implementations for that matter.
AMQP
• AMQP Working Group (Community and Vendor)
• Platform agnostic protocol.
• Completely open, interoperable and broadly applicable.
• Many severs available and many client libraries.
STOMP
• Simple protocol
• Behaviors follow very simple commands.
• Most message queues can communicate over STOMP.
Connect Send Disconnect
/queue/msg
PHP
STOMP
SERVER
Connect Subscribe Disconnect
/queue/msg
Read
Ack
SQS
• Fairly simple protocol
• Supports delays, timers, and multiple policies.
SPECIAL PURPOSE
Many queue implementations exist that don’t necessarily sit
under standards…
XMPP
• Best for real-time data.
• Leveraging pub/sub can turn it into more of a generic message system.
• Multiple libraries available.
SOCKET.IO
• New comer
• Real-time bidirectional event-based communication
• Largely leverages pub/sub
ZEROMQ
• The ultimate in message queue flexibility.
• Socket library that acts as a concurrency framework.
GEARMAN
• Application framework for farming out work.
• Job sever for asynchronous or synchronous messages.
BEANSTALKD
• Asynchronous Job Queue
• Supports delays
• Many PHP clients exist
CONSIDERATIONSHow do we evaluate our options…
PULL VS. PUSH
• Always PULL, whenever possible.
• Push eliminates several benefits.
DURABILITY
• Memory residence
• Persistence
• Restart survival
SECURITY
• Authentication
• Queue permissions / restrictions
DELIVERY
• Is the delivery guaranteed?
• If a message cannot be delivered how it it handled?
ROUTING
• Multiple routing scenarios
• Fanout
• Direct
• Topic
• Broadcast
BATCHING
• Do it later but in bulk (credit card processing)
• Can be done via scheduling (cron)
RECEIPT
• Do you get an acknowledgement of receipt?
IMPLEMENTING QUEUES
STARTING POINTS
PUSHING MESSAGES<?php
class UserService { public function save($user) { $this->db->save($user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', [ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); } }
HANDLING MESSAGES
<?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email');
while (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body);
mail($email->to, $email->subject, $email->message, $email->headers); }
MESSAGES
MESSAGE CONSIDERATIONS
• Message Format
• Message Contents
SERIALIZE
O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:3:"baz";}}}}
WORKERS
WORKER CONSIDERATIONS
• Should do ONE thing and ONE thing well.
• Should attempt to be as quick as possible in handling that type.
• Should be able to be scaled horizontally.
HANDLING WORKERS
• Prevent Memory Leaks
• memory_get_usage
• Handle Signals!
• pcntl_signal
ABSTRACTIONSWe want to make this easy…
<?php
interface QueueInterface { public function __construct(Stomp $stomp, $queue); public function dispatch(); public function publish(array $message); public function work(StompFrame $message); }
<?php
class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected $signal;
public function __construct(Stomp $stomp, $queue) { $this->stomp = $stomp; $this->queue = $queue; }
protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); }
if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } }
protected function signal($signal) { $this->signal = $signal; }
public function dispatch() { $this->prepare();
while (true) { if ($this->signal) { break ; }
if (!$this->stomp->hasFrame()) { $this->wait(); continue ; }
$frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); }
$this->stomp->ack($frame); } }
protected function wait() { sleep(1); }
protected function validate(StompFrame $message) { return false; }
public function publish(array $message) { return $this->stomp->send($this->queue, json_encode($message)); }
<?php
class EmailQueue extends AbstractQueue { public function validate(StompFrame $message) { if (!array_key_exists('to', $message)) { return false; } return true; }
public function work(StompFrame $message) { $mail = json_decode($message);
mail($mail->to, $mail->subject, $mail->message); } }
BOOTSTRAPPINGLeverage your existing infrastructure as much as possible!
<?php declare(ticks=1);
include 'vendor/autoload.php'; $app = Zend\Mvc\Application::init(include 'config/application.config.php'); $sm = $app->getServiceManager();
if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>\n\n"); exit(1); } $name = $argv[1];
try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null;
echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);
EVENTS
SERVICES TRIGGER EVENTS
<?php
use Zend\EventManager\EventManagerAwareTrait;
class UserService { use EventManagerAwareTrait;
public function save($user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }
ATTACH EVENTS
<?php
use Zend\ServiceManager\ServiceManager;
$sm = new ServiceManager(); $service = $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish([ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });
HANDLING PROGRESS
• Keep track by using a generic handler
• Or, keep track via your database.
TOOLING
SUPERVISOR
• Daemon that runs on the server.
• Monitors programs and keeps them running in case of failure.
• Handles logging.
PAINLESS INSTALLATION
sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo service supervisor start
EXAMPLE PROGRAM CONFIGURATION
[program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email"
process_name=%(program_name)s_%(process_num)d numprocs=2
numprocs_start=2 user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15
SUPERVISORD MULTI SERVER MONITORING TOOL
https://github.com/mlazarov/supervisord-monitor
WHEN BAD THINGS HAPPEN
QUEUE BACKUP
• Most of all issues with queues are that a queue has backed up.
WORKER EXCEPTIONS
• Broken code causes hardships, but recoverable!
• https://pixabay.com/en/autobahn-accident-germany-car-road-837643/• https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/• https://pixabay.com/en/airplanes-line-runway-military-713662/• https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/• https://pixabay.com/en/user-top-view-office-keyboard-154199/• https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/• https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/• https://pixabay.com/en/umbrellas-red-blue-patterns-205386/• https://pixabay.com/en/spot-runs-start-la-stadion-862274/• https://pixabay.com/en/artistic-the-art-of-abstraction-948588/• https://pixabay.com/en/boots-work-boots-shoes-647035/• https://pixabay.com/en/meerkat-watch-guard-cute-676944/• https://pixabay.com/en/broken-window-hole-glass-damage-960188/• https://pixabay.com/en/police-security-safety-protection-869216/• https://pixabay.com/en/parcel-package-packaging-box-575623/• https://pixabay.com/en/directory-signposts-trail-direction-494457/• https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/• https://pixabay.com/en/phone-communication-call-select-735060/• https://pixabay.com/en/receipt-note-paper-bill-document-575750/• https://pixabay.com/en/tools-construct-craft-repair-864983/• https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/• https://pixabay.com/en/network-iot-internet-of-things-782707/• https://pixabay.com/en/mobile-phone-smartphone-app-426559/• https://pixabay.com/en/mr-site-build-crane-baukran-462074/• https://pixabay.com/en/film-projector-movie-projector-738806/• https://pixabay.com/en/calves-legs-human-standing-on-540519/• https://pixabay.com/en/notebook-pages-opened-paper-note-820078/• https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/• https://pixabay.com/en/spray-household-surface-shine-315164/• https://pixabay.com/en/industry-crafts-gears-mechanical-94448/• https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/• https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/• https://pixabay.com/en/temple-china-door-handle-840526/• https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/
Image Credits
THANK YOU!http://joind.in/talk/view/15538