Eugene Shakhsuvarov - Improving enterprise store scalability using AMQP and Asynchronous Operations


Improving Enterprise Store Scalability with Queues


Message Queue Framework in Magento EEOverview of capabilities


What are Queue advantages?• Execute tasks in background process• Asynchronously, with multiple workers• Take advantage of RabbitMQ benefits:

– High Availability– Scalability– Load Balancing– Postponed execution– Platform Agnostic– Open Source


What does Magento EE add to this?• Provides support for queuing with the

simple API• Transparent workflow• Takes care of topology with just xml

configurations• Great way to eliminate blocking

requests• Fallback queue solution based solely

on MqSQL for testing and specific applications


Basic Concepts• Message is an atomic unit of processing• Message exists in queue until explicitly deleted• Never sent more then once

• Always sent with specific topic• Consumers get appropriate messages by

subscribing to the desired topic


Basic Workflow


Publisher• Publishes message data to the specific exchange• Data may be represented as an arbitrary array or DTO• Configured with xml configuration• Simple interface


Consumer• Consumes message from the queue• Passes it to the configured handler class for processing• Again, simple xml configuration


Batch Consumer• For the cases where lots of messages are involved• Consumes a batch of multiple messages of configured size• Allows handler to process whole batch in a loop


Configuration• Queues configuration happens in a few files inside the module’s

“etc” directory.• “communication.xml” is responsible for aspects of the message

queue system that all communication types have in common. AMQP and Database connections. Values may be overridden for deployment in “env.php”.

• “queue.xml” is responsible for defining the broker that processes the topic.

• “queue_consumer.xml” responsible for configuring a consumer, allowing to set it’s type and handler for messages.

• “queue_publisher.xml” configures publisher connection to the message broker.


Are there any disadvantages?• Concurrent execution of similar actions

may lead to locks in specific scenarios• No guarantee on the timeframe, when

the task will be processed

• Generally, using a single worker decreases overall performance:– Using Multiple workers improves

performance, but only up to a certain grade.


Asynchronous Operations ModuleYes, it gets even better.


Asynchronous Operations in Magento EE• Significantly improves “Look and Feel” for the merchant• Allows Store Admin to send entity updates to the queue for

background processing eliminating long blocking requests• Provides a UI for tracking status of the execution and logging• Allows to retry certain failed operations• Built on top of the Message Queue Framework


Asynchronous Operations


Magento Asynchronous Operations Workflow• Create Operation per entity to be processed• Form bulk of operations• Post messages to the Queue• Handle operations and their statuses from the Queue

Operation

Operation

Operation

Queue Handler UI


Operation• Operation is a single unit of work to be

processed• Framework interprets it as a simple DTO

containing:– Operation UUID– Name of the destination topic in Queue– Serialized data or DTO for processing– Operation Status and Status Message

• Recoverably Failed operations may be retried in a separate bulk


Operation Statuses• Operations can be in one of the 4 possible statuses:

– Open: still in queue– Complete: processed successfully without errors– Failed Recoverably: based on exception type it is possible to determine

if operation is worth retrying. For example a connection, wait or lock exceptions, which, in some cases, may happen when dealing with the large number of similar items

– Failed Non Recoverably: for cases when operation is failed due to reasons which require manual interaction, for example validation errors

• Operation status is accompanied by Result Message, visible to the store admin, which may contain more detailed information on the problem


Bulk Management• Bulk is an array of operations,

identified by Unique ID• Visible as a separate entry to the

admin user who created it, along with the count and status of operations within it

• Every Operation is published as a separate message under a desired topic name

• Bulk Management provides a simple API to publish Operations into the Queue


How does it look for a store admin?


Quick Recap


Quick recap• Queue is good for performing long operations in background• Great way to avoid blocking requests• Scales well with the number of workers, however has its limits• Simple API on Magento's side


Example Module• You may take a look at the example module, which using some of

the discussed features updates Catalog Product Stock Quantities

• https://tinyurl.com/mmit17-queues• https://github.com/ishakhsuvarov/mmit17-queue-example


Q&AFor more questions: @ishakhsuvarov(twitter, github or @magento.com email)

Eugene Shakhsuvarov - Improving enterprise store scalability using AMQP and Asynchronous Operations

Presentations & Public Speaking

Transcript of Eugene Shakhsuvarov - Improving enterprise store scalability using AMQP and Asynchronous Operations