Massaging the Pony: Message Queues and You
-
Upload
shawn-rider -
Category
Technology
-
view
3.520 -
download
2
description
Transcript of Massaging the Pony: Message Queues and You
![Page 1: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/1.jpg)
Massaging the Pony:Message Queues and You
Shawn RiderPBS Education
DjangoCon 2010
http://www.flickr.com/photos/funtik/1175522045
![Page 2: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/2.jpg)
What is a Message Queue?1. A system for enabling asynchronous
processing of discrete tasks.2. Super awesome.
http://www.flickr.com/photos/gadl/89650415
![Page 3: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/3.jpg)
What are they good for?
Alleviate server issues by offloading processing
http://www.flickr.com/photos/islandgyrl/3197469932
![Page 4: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/4.jpg)
What are they good for?
Make better user experiences
http://www.flickr.com/photos/bbaltimore/6779682
![Page 5: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/5.jpg)
What are they good for?
Increase the reliability of third party service integrations.
![Page 6: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/6.jpg)
What are they good for?
Background media processinghttp://www.flickr.com/photos/factoryjoe/2882992091
![Page 7: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/7.jpg)
What are they good for?
• Repeating Tasks can replace Cron jobs • Keep all your code in your project
![Page 8: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/8.jpg)
Available MQ Solutions
There are many solutions out there. To name a few of them:
• RabbitMQ• Amazon Simple Queue System• Apache MQ• Gearman• OpenAMQ• Peafowl• Q4M• Starling• Sun Java System Message Queue
![Page 9: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/9.jpg)
Message Queue Protocols
• AMQP: Advanced Message Queuing Protocol• JMS: Java Messaging Service• STOMP: Streaming Text Oriented Messaging
Protocol
![Page 10: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/10.jpg)
Criteria for Broker Selection
The following criteria were important to us in selecting a solution. You may have some different needs.
• Handling exceptions and recovery• Low level of required maintenance• Ease of deployment• Persistent queuing• Community support• What language is it written in? How compatible is that with
our current systems?• Cluster support
![Page 11: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/11.jpg)
Failsafe MQ Brokers
• Must be able to survive a server failure.• Upon restarting the MQ Broker, processing
should resume from the last un-processed messages in queues
![Page 12: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/12.jpg)
Queue Durability
The Messages Queue Broker must be able to reconstruct the queues when it is restarted.
http://www.flickr.com/photos/raul/846318014
![Page 13: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/13.jpg)
Message Persistence
The Messages (tasks to be completed) must persist between shutdowns and restarts of MQ Server
![Page 14: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/14.jpg)
Our Choice
• Celery (a Django task management app)• Carrot (Django middleware)• PyAMQP (Python module)• RabbitMQ (MQ Broker software)
![Page 15: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/15.jpg)
The Production System
Webapp
DatabaseWorker
Server(s)
CeleryRabbit
MQ
![Page 16: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/16.jpg)
MQ in Development Environs
It is important to be able to develop Message Queue tasks in Django without being concerned about running the broker process, queue setup, persistence etc.
![Page 17: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/17.jpg)
MQ in Development Environs
• Carrot can be set up to use different queuing backend for development purposes
• GhettoQ provides database backend for carrot where a database serves as the queue storage
• The broker is a simple Django application that monitors the queue in DB
• Inefficient, therefore only suitable for development environments
CARROT_BACKEND = ‘qhettoq.taproot.Database’
![Page 18: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/18.jpg)
Ease a future MQ implementation
Isolate functionality into reusable components.
![Page 19: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/19.jpg)
Recycle Existing Code
![Page 20: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/20.jpg)
Django, HTTP Requests & MQ
• Most functionalities provided through Django applications are tightly coupled with HTTP requests e.g. form data processing
• In order to asynchronously execute a task, Celery (any task management platform) must serialize & store parameters
• HTTP requests are usually large and not easily serializable. e.g. WSGI requests
![Page 21: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/21.jpg)
The PickleableHTTPRequest Object
![Page 22: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/22.jpg)
Using PickleableHttpRequest
if request.GET.get(‘download’): try: result = EnrollmentCSVExportTask.delay( PickleableHttpRequest( request, attributes=\ [‘user’, ‘admin_current_organization’] ) )
![Page 23: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/23.jpg)
Making Message Queue Tasks
It is fast and easy to add a decorator to specific functions to create Message Queue tasks.
from celery.decorators import task
@taskdef add(x, y): return x + y
![Page 24: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/24.jpg)
Explicitly Inherit from Celery’s Task Object
![Page 25: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/25.jpg)
Then What?
Some Message Queue tasks can complete with no notification required.
For other tasks:• Notify by email• Use an AJAX listener to notify users• Some other messaging
![Page 26: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/26.jpg)
In Review
• Choose your MQ solution wisely• Set up a robust system• Leverage your existing code• Replace trouble spots as they come up• Profit!
![Page 27: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/27.jpg)
Message Queues Make Life Better
![Page 28: Massaging the Pony: Message Queues and You](https://reader035.fdocuments.us/reader035/viewer/2022062703/5552c156b4c905920f8b48b6/html5/thumbnails/28.jpg)
References and Links
Second Life’s Survey of Message Queues:http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes
Ask Solem Hoel’s Github Repos: http://github.com/askCelery Project: http://celeryproject.org/
PickleableHttpRequest by Nowell Strite (@nowells)http://gist.github.com/571027
Special thanks to Tareque Hossain (http://codexn.com)