Scaling applications with RabbitMQ at SunshinePHP
149
Scaling Applications with RabbitMQ Alvaro Videla - RabbitMQ
-
Upload
alvaro-videla -
Category
Technology
-
view
3.249 -
download
3
description
Do you need to process thousands of images in the background for your web app? Do you need to share data across multiple applications, probably written in different languages and sitting at different servers? Your real time data feed is becoming slow because you are polling the database constantly for new data updates? Do you need to scale information processing during peek times? What about deploying new features with zero downtime? If any of these problems sound familiar then you probably need to use messaging in your application. In this talk I will introduce RabbitMQ, a messaging and queue server that can help us tackle those problems. We will learn the benefits of a Queue Server and see how to integrate messaging into our applications. With this talk we hope that the term 'decoupling' gets a new, broader, meaning.
Transcript of Scaling applications with RabbitMQ at SunshinePHP
Scaling Applications with RabbitMQ
Alvaro Videla - RabbitMQ
Alvaro Videla
• Developer Advocate at Pivotal / RabbitMQ!
• Co-Author of RabbitMQ in Action!
• Creator of the RabbitMQ Simulator!
• Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html!
• @old_sound — [email protected] — github.com/videlalvaro
About this Talk
• Exploratory Talk
• A ‘what could be done’ talk instead of ‘this is how you do it’
What is RabbitMQ
RabbitMQ
RabbitMQ
RabbitMQ• Multi Protocol Messaging Server
• Multi Protocol Messaging Server!• Open Source (MPL)
RabbitMQ
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot
RabbitMQ
• Multi Protocol Messaging Server!• Open Source (MPL)!• Polyglot!• Written in Erlang/OTP
RabbitMQ
Multi Protocol
http://bit.ly/rmq-protocols
Polyglot
Polyglot
Polyglot
• Java
Polyglot
• Java!• node.js
Polyglot
• Java!• node.js!• Erlang
Polyglot
• Java!• node.js!• Erlang!• PHP
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net
Polyglot
• Java!• node.js!• Erlang!• PHP!• Ruby!• .Net!• Haskell
Polyglot
Even COBOL!!!11
Some users of RabbitMQ
Some users of RabbitMQ
• Instagram!• Indeed.com
Some users of RabbitMQ
• Instagram!• Indeed.com!• Telefonica
Some users of RabbitMQ
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre
Some users of RabbitMQ
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS
Some users of RabbitMQ
• Instagram!• Indeed.com!• Telefonica!• Mercado Libre!• NHS!• Mozilla
Some users of RabbitMQ
http://www.rabbitmq.com/download.html
Unix - Mac - Windows
Messaging with RabbitMQ
A demo with the RabbitMQ Simulator
https://github.com/RabbitMQSimulator/RabbitMQSimulator
RabbitMQ Simulator
What you get out of the box
Some RabbitMQ Features
Some RabbitMQ Features• No message loss
Some RabbitMQ Features• No message loss
• Persistent Messages
Some RabbitMQ Features• No message loss
• Persistent Messages
• Publisher Confirms
Some RabbitMQ Features• No message loss
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
Some RabbitMQ Features• No message loss
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
Some RabbitMQ Features• No message loss
• Persistent Messages
• Publisher Confirms
• Message Acknowledgment
• Mirrored Queues
• Message Ordering
Some RabbitMQ Features
Some RabbitMQ Features• Dead Letter Exchanges
Some RabbitMQ Features• Dead Letter Exchanges
• Alternate Exchanges
Some RabbitMQ Features• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
Some RabbitMQ Features• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
Some RabbitMQ Features• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
Some RabbitMQ Features• Dead Letter Exchanges
• Alternate Exchanges
• Message and Queues TTLs
• Consumer Priorities
• Federation
• Shovel
RabbitMQ Management
Scenario
Batch Processing
Requirements
Requirements
• Generate XML
Requirements
• Generate XML
• Distribution Over a Cluster
Requirements
• Generate XML
• Distribution Over a Cluster
• Elasticity - Add/Remove new workers
Requirements
• Generate XML
• Distribution Over a Cluster
• Elasticity - Add/Remove new workers
• No Code Changes
Design
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Publisher Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);!$msg = new AMQPMessage($video_info,
array('content_type' => 'text/plain', 'delivery_mode' => 2));
!$channel->basic_publish($msg, 'video-desc-ex');!$channel->close();$conn->close();
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Consumer Code
$conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST);$channel = $conn->channel();!$channel->exchange_declare('video-desc-ex', 'direct', false,
true, false);$channel->queue_declare('video-desc-queue', false, true,
false, false);$channel->queue_bind('video-desc-queue', 'video-desc-ex');!$channel->basic_consume('video-desc-queue', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) { $channel->wait();}
Scenario
Upload Pictures
Requirements
Requirements
• Upload Picture
Requirements
• Upload Picture
• Reward User
Requirements
• Upload Picture
• Reward User
• Notify User Friends
Requirements
• Upload Picture
• Reward User
• Notify User Friends
• Resize Picture
Requirements
• Upload Picture
• Reward User
• Notify User Friends
• Resize Picture
• No Code Changes
Design
Design
Design
Publisher Code$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);!$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
!$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));!$channel->basic_publish($msg, 'upload-pictures');
Publisher Code$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);!$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
!$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));!$channel->basic_publish($msg, 'upload-pictures');
Publisher Code$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);!$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
!$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));!$channel->basic_publish($msg, 'upload-pictures');
Publisher Code$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);!$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
!$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));!$channel->basic_publish($msg, 'upload-pictures');
Publisher Code$channel->exchange_declare('upload-pictures', 'fanout', false, true, false);!$metadata = json_encode(array('image_id' => $image_id,'user_id' => $user_id,‘image_path' => $image_path));
!$msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2));!$channel->basic_publish($msg, 'upload-pictures');
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code$channel->exchange_declare('upload-pictures', 'fanout',
false, true, false);!$channel->queue_declare('resize-picture', false, true,
false, false);!$channel->queue_bind('resize-picture', 'upload-pictures');!$channel->basic_consume('resize-picture', $consumer_tag,
false, false, false, false, $consumer);!while(count($channel->callbacks)) {$channel->wait();
}
Consumer Code
$consumer = function($msg){!$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
Consumer Code
$consumer = function($msg){!$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
Consumer Code
$consumer = function($msg){!$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
Consumer Code
$consumer = function($msg){!$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
Consumer Code
$consumer = function($msg){!$meta = json_decode($msg->body, true);
resize_picture($meta['image_id'], $meta['image_path']);
$msg->delivery_info['channel']->basic_ack($msg->delivery_info['delivery_tag']);
};
Scenario
Distributed Logging
Requirements
Requirements• Several Web Servers
Requirements• Several Web Servers
• Logic Separated by Module/Action
Requirements• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels: info, warning and error
Requirements• Several Web Servers
• Logic Separated by Module/Action
• Several Log Levels: info, warning and error
• Add Remove log listeners at will
Design
Design
Design
Design
Design
Publisher Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));!$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');
Publisher Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));!$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');
Publisher Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));!$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');
Publisher Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$msg = new AMQPMessage('some log message',
array('content_type' => 'text/plain'));!$channel->basic_publish($msg, 'logs',
'server1.user.profile.info');
Consumer Code
Get messages sent by host: !
server1
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);!
$channel->queue_declare('server1-logs', false, true, false, false);!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);!
$channel->queue_declare('server1-logs', false, true, false, false);!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);!
$channel->queue_declare('server1-logs', false, true, false, false);!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);!
$channel->queue_declare('server1-logs', false, true, false, false);!
$channel->queue_bind('server1-logs', 'logs', 'server1.#');
Consumer Code
Get all error messages
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$channel->queue_declare('error-logs', false, true,
false, false);!
$channel->queue_bind('error-logs', 'logs', '#.error');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$channel->queue_declare('error-logs', false, true,
false, false);!
$channel->queue_bind('error-logs', 'logs', '#.error');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$channel->queue_declare('error-logs', false, true,
false, false);!
$channel->queue_bind('error-logs', 'logs', '#.error');
Consumer Code
$channel->exchange_declare('logs', 'topic', false, true, false);
!$channel->queue_declare('error-logs', false, true,
false, false);!
$channel->queue_bind('error-logs', 'logs', '#.error');
Distributed Apps
Distributed Application
App
App
App
App
Distributed Application
App
App
App
App
Ad-hoc solution
A process that replicates data to the remote server
Possible issues• Remote server is offline
• Prevent unbounded local buffers
• Prevent message loss
• Prevent unnecessary message replication
• No need for those messages on remote server
• Messages that became stale
Can we do better?
RabbitMQ Federation
RabbitMQ Federation• Supports replication across different administrative domains
• Supports mix of Erlang and RabbitMQ versions
• Supports Network Partitions
• Specificity - not everything has to be federated
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation
RabbitMQ Federation
• It’s a RabbitMQ Plugin
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
RabbitMQ Federation
• It’s a RabbitMQ Plugin
• Internally uses Queues and Exchanges Decorators
• Managed using Parameters and Policies
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
Enabling the Plugin
rabbitmq-plugins enable rabbitmq_federation
rabbitmq-plugins enable rabbitmq_federation_management
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}'
Federating an Exchange
rabbitmqctl set_parameter federation-upstream my-upstream \ ‘{“uri":"amqp://server-name","expires":3600000}' !
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\." \ '{"federation-upstream-set":"all"}'
Federating an Exchange
With RabbitMQ we can
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using with different consumer strategies
With RabbitMQ we can
• Ingest data using various protocols: AMQP, MQTT and STOMP
• Distribute that data globally using Federation
• Scale up using with different consumer strategies
• Integrate across many platforms and programming languages
Questions?
https://joind.in/talk/view/10525
Talk and Slides
ThanksAlvaro Videla - @old_sound
world map: wikipedia.org federation diagrams: rabbitmq.com
Credits
