Do More With Message Queue
-
Upload
hean-hong-leong -
Category
Software
-
view
169 -
download
0
Transcript of Do More With Message Queue
![Page 1: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/1.jpg)
Do More With Message Queue
Leong Hean Hong (mrleong.net)2nd Aug, 2015
![Page 2: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/2.jpg)
Objective
Optimizing web application using message queue
![Page 3: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/3.jpg)
Agenda
● Problem● Solution● Message Queue● Queue Systems● Protocols● Benchmark● Choosing Queue System● Example
![Page 4: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/4.jpg)
Problem
● An author publishes an article on your website
● The website needs to ○ send push notifications to 2000 followers○ send email notification to 2000 followers○ optimise images in the article○ announce it on various (social) media
When user clicked ‘Submit’ button, how long does she have to wait?
![Page 5: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/5.jpg)
Solution
● Show a publish success message to user immediately
● Schedule the tasks to be performed later● Perform multiple tasks concurrently
![Page 6: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/6.jpg)
Queue your tasks
![Page 7: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/7.jpg)
Message Queue
![Page 8: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/8.jpg)
Producer
● Request for tasks to be performed● Tasks may not be performed immediately
upon request
![Page 9: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/9.jpg)
Message
● Contains information about a task● Information must be understood by task
processor (consumer)
![Page 10: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/10.jpg)
Message Broker
● Manage multiple queues● Receive messages and store in queue(s)● Allow consumer to retrieve messages
![Page 11: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/11.jpg)
Consumer
● Retrieve message from message broker● Process the task● Producer and consumer might be the same
application
![Page 12: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/12.jpg)
● RabbitMQ● ActiveMQ (most popular)● SQS (Amazon Simple Queue Service)● ZeroMQ (decentralized, blazing fast)● HornetQ● Apollo● QPID
Reference: http://queues.io/
Queue Systems
![Page 13: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/13.jpg)
Protocols
● AMQP○ Advanced Message Queuing Protocol○ Most popular
● STOMP○ Streaming Text Oriented Messaging Protocol
![Page 15: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/15.jpg)
Choosing Queue System
● Persistent / non-persistent● Setup/integration effort● (Community) support● Licensing cost● Maintenance effort/cost● Performance
![Page 16: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/16.jpg)
Example
Custom made message forwarder: https://github.com/hongster/message-forwarder
![Page 17: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/17.jpg)
Example
1. Web app sends a message to RabbitMQ server.a. Message contains worker’s URL in custom AMQP
headerb. Message content might be ID of posted article
2. We developed an app (Message Forwarder) that constantly a. get message from RabbitMQ serverb. forwards message to worker using HTTP POSTc. multiple messages are sent concurrently
![Page 18: Do More With Message Queue](https://reader036.fdocuments.us/reader036/viewer/2022071910/55ce98d3bb61eb40628b461a/html5/thumbnails/18.jpg)
Example
● Worker is another web app that do work upon receiving message.○ Sometime it is the the original sender web app itself
● Worker might be a webservice, and URL is the API endpoint