Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product...
Transcript of Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product...
![Page 1: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/1.jpg)
© 2018 Rock Solid Knowledge.
Messaging and RabbitMQ
![Page 2: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/2.jpg)
© 2018 Rock Solid Knowledge 2
Agenda
• The problem space
• Why queues?
• Reliability
• Failure Semantics
• Scalability and Availability
![Page 3: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/3.jpg)
© 2018 Rock Solid Knowledge 3
Terminology
• Broker
• Overall controller of messaging infrastructure
• Queue
• FIFO structure that stores messages
• Publisher
• A part of a system that creates messages and sends to a queue (also known as a producer)
• Consumer
• A part of a system that reads messages from a queue and performs any resultant processing
![Page 4: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/4.jpg)
© 2018 Rock Solid Knowledge 4
Why Queues?
• Decouple publisher and consumer
• Fully async
• Don’t have to be running concurrently
Publisher Consumer
Publisher’s World Consumer’s World
![Page 5: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/5.jpg)
© 2018 Rock Solid Knowledge 5
RabbitMQ
• Open Source Queuing Product
• Written in Erlang
• Designed for high performance
• Google have installation processing 1,000,000 message/second
• Flexible for most situations
• May require a little work
• .NET library on Nuget
• RabbitMQ.Client
![Page 6: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/6.jpg)
© 2018 Rock Solid Knowledge 6
Connections and Models
• Access RabbitMQ via a connection
• Authentication details
• Server (broker) address
• Perform actions via a model
• Recommended 1 model per thread
ConnectionFactory factory = new ConnectionFactory{
HostName = host,UserName = user,Password = password
};
IConnection connection = factory.CreateConnection();
IModel model = connection.CreateModel();
![Page 7: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/7.jpg)
© 2018 Rock Solid Knowledge 7
Publishing via Exchanges
• Publish to Exchange rather than queue
• Queues bound to exchanges via routing criteria
• Level of indirection increases flexibility
• Publication specifies a routing key for the message
• Determines routing matches for queues
• Routing matching rules depend on type of exchange
model.ExchangeDeclare(exchangeName, "direct", false, false);
string bodyMessage = "Hello, World!";byte[] body = Encoding.UTF8.GetBytes(bodyMessage);
model.BasicPublish(exchangeName, routingKey, false, null, body);
![Page 8: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/8.jpg)
© 2018 Rock Solid Knowledge 8
Messages
• Messages very flexible in RabbitMQ
• Can have body
• Can have headers
• Can modify behaviour with properties
![Page 9: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/9.jpg)
© 2018 Rock Solid Knowledge 9
Queues and Bindings
• Queues define where messages are stored
• Binding determines which queues receive a message when published
• All matches receive copy
• No match may or may not be an issue
model.ExchangeDeclare(exchangeName, "direct", false, false);
model.QueueDeclare(queueName, false, false, false, null);
model.QueueBind(queueName, exchangeName, routingKey, null);
![Page 10: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/10.jpg)
© 2018 Rock Solid Knowledge 10
Consuming a queue
• Pass IBasicConsumer to model
• RabbitMQ library provides EventingBasicConsumer
• Each message arrives as an event
• Two consumers on same queue will not get same message
var consumer = new EventingBasicConsumer(model);
consumer.Received += (s, e) =>{
string bodyMessage = Encoding.UTF8.GetString(e.Body);
Console.WriteLine(bodyMessage); };
model.BasicConsume(queueName, true, consumer);
![Page 11: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/11.jpg)
© 2018 Rock Solid Knowledge 11
Load Leveling
• Queues provide load leveling automatically
• Consumer does not have to be able to process peak load
• Handling average load is enough unless processing latency is an issue
![Page 12: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/12.jpg)
© 2018 Rock Solid Knowledge 12
Messaging Patterns
• Can implement different messaging patterns in RabbitMQ
• Pub/Sub
• Broadcast
• Async Request/Response
• Different types of exchanges assist
• Direct
• Topic
• Header
• Fanout
![Page 13: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/13.jpg)
© 2018 Rock Solid Knowledge 13
Direct Exchange
• Published message routingKey must exact match routingKey of queue binding
• Most performant
model.ExchangeDeclare(exchangeName, "direct", false, false);
model.QueueDeclare(queueName, true, false, false, null);
model.QueueBind(queueName, exchangeName, routingKey, null);
model.BasicPublish(exchangeName, routingKey, true, props, null);
![Page 14: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/14.jpg)
© 2018 Rock Solid Knowledge 14
Topic Exchange
• Published message routingKey pattern matched against routingKey of queue binding
• Pub/Sub common use case
• Routing key typically set of dot separated segments
• JobType.SalesReport.AcmeCorp
model.ExchangeDeclare(exchangeName, "topic", false, false);
model.QueueDeclare(queueName, true, false, false, null);
model.QueueBind("reportQueue", exchangeName, "JobType.SalesReport.#", null);
model.BasicPublish(exchangeName, routingKey, true, props, null);
![Page 15: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/15.jpg)
© 2018 Rock Solid Knowledge 15
Headers Exchange
• Published message headers must match headers specification of queue binding
• Can match on one or more headers
• Match type all or any
model.ExchangeDeclare(exchangeName, "headers", false, false);
model.QueueDeclare(queueName, true, false, false, null);
var spec = new Dictionary<string, object>{
["priority"] = 10,["retryAttempts"] = 2,["x-match"] = "all"
};
model.QueueBind(”reportQueue", exchangeName, "", spec);
![Page 16: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/16.jpg)
© 2018 Rock Solid Knowledge 16
Fanout Exchange
• Published message delivered to all bound queues irrespective of routingKey
• Used for broadcast
FanoutExchange
![Page 17: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/17.jpg)
© 2018 Rock Solid Knowledge 17
Queues and Reliability
• Messages can be persisted
• Systems can recover easily after crashes
• Must be managed by message broker
Publisher Consumer
Publisher’s World Consumer’s World
![Page 18: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/18.jpg)
© 2018 Rock Solid Knowledge 18
RabbitMQ, Persistence and Durability
• Durability and Persistence are often confused in RabbitMQ
• Durability is about exchanges and queues – if the broker restarts are they automatically recreated?
• Persistence is about messages – if a machine restarts are the messages still available?
![Page 19: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/19.jpg)
© 2018 Rock Solid Knowledge 19
Durability
• Durable flag when declaring exchanges and queues
• Exchange/Queue will be recreated if broker restarts
• Can also mark an exchange/queue as auto-delete
• Will be deleted when last connection disconnects
• Also flag on declaration
model.ExchangeDeclare(exchangeName, "direct", true, false);
model.QueueDeclare(queueName, true, false, false, null);
model.ExchangeDeclare(exchangeName, "direct", true, false);
model.QueueDeclare(queueName, true, false, false, null);
![Page 20: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/20.jpg)
© 2018 Rock Solid Knowledge 20
Persistence
• Message can be persisted
• Queue must be durable
• Performance implications (1 – 2 orders of magnitude slower)
• Specify DeliveryMode = 2 in message properties
var props = model.CreateBasicProperties();
props.DeliveryMode = 2; // persistent message
model.BasicPublish(exchangeName, routingKey + "." + suffix, true, props, null);
![Page 21: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/21.jpg)
© 2018 Rock Solid Knowledge 21
Reliability (cont’d)
• Need to ensure message published and message processed
• Traditionally done through transactions
Publisher Consumer
Tx1 Tx2
![Page 22: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/22.jpg)
© 2018 Rock Solid Knowledge 22
RabbitMQ and Reliability
• RabbitMQ supports transactions but discouraged
• Heavy performance impact
• Instead RabbitMQ typically uses confirmations and acknowledgements
• Receiver of message must state success (Ack) or failure (NAck)
• Broker reports to publisher
• Consumer reports to broker
![Page 23: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/23.jpg)
© 2018 Rock Solid Knowledge 23
Tracking acknowledgements
• Every message published or received on a model is given a model specific numeric id
• DeliveryTag
• Incremented by one for each message
• Success or failure reported for each id
• Can report multiple successes or failures in one round trip
• Important for high volume systems
![Page 24: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/24.jpg)
© 2018 Rock Solid Knowledge 24
autoAck
• By default consumer uses automatic acknowledgements (autoAck)
• Message Ack’d once received but before processed so message can be lost
• Can disable autoAck in BasicConsume
• Consumer must now manually ack
model.BasicConsume(queueName, false, consumer);
consumer.Received += (s, e) => {try {
ProcessMessage(e.Body);model.BasicAck(e.DeliveryTag, false);
}catch (Exception exception) {
model.BasicNack(e.DeliveryTag, false, true);}
};
![Page 25: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/25.jpg)
© 2018 Rock Solid Knowledge 25
Publishing with Confirms
• Did the broker receive the message OK?
• Can turn on confirmations
• Broker fires model events: BasicAcks or BasicNacks
• DeliveryTag in event args identifies message
• May get multiple confirms in one event
model.ConfirmSelect();
model.BasicAcks += (s, e) =>{
// ...};
model.BasicNacks += (s, e) =>{
// ...};
![Page 26: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/26.jpg)
© 2018 Rock Solid Knowledge 26
Routing Failures
• What if message cannot be routed to a queue?
• Default to drop on floor
• Often valid for pub/sub scenarios
• Can specify the successful routing is mandatory
model.BasicPublish(exchangeName, routingKey, true, props, null);
![Page 27: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/27.jpg)
© 2018 Rock Solid Knowledge 27
Failed mandatory routing
• Broker returns message to publisher using BasicReturn
• Need someway to match message publication to message return
• Message not identified with DeliveryTag
• Can create messageId header to track
![Page 28: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/28.jpg)
© 2018 Rock Solid Knowledge 28
Failure Semantics
• What happens if I fail to publish to queue?
• Depends on queue implementation
• What happens if I fail to process a message?
• Drop on floor?
• Requeue?
Consumer
Poison Message?
Poison Message Queue
![Page 29: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/29.jpg)
© 2018 Rock Solid Knowledge 29
RabbitMQ and Poisoned Messages
• Can configure dead letter exchange on queue
• Message sent to dead letter exchange if message NAck’d with requeue false
var args = new Dictionary<string, object>{
["x-dead-letter-exchange"] = "DeadLetters"};
model.QueueDeclare(queueName, true, false, false, args);
![Page 30: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/30.jpg)
© 2018 Rock Solid Knowledge 30
Is message poisoned?
• No built in poison message identification
• Cannot tell how many times a message has attempted processing
• Can keep track out of band if using unique message id
• Can add retry count into header – means manual requeuing rather than NAck
![Page 31: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/31.jpg)
© 2018 Rock Solid Knowledge 31
Security and Virtual Hosts
• RabbitMQ uses Virtual Hosts for isolation
• Exchanges and queues live in a virtual host
• Default virtual host is “/”
• User account authorization based on virtual host
• Configure
• Write
• Read
• Pattern matched to artifacts using regex
![Page 32: Messaging and RabbitMQ - SDD Conference · 2019-05-24 · RabbitMQ • Open Source Queuing Product • Written in Erlang • Designed for high performance • Google have installation](https://reader034.fdocuments.us/reader034/viewer/2022050603/5faac8cbe63b6404bf4d2be8/html5/thumbnails/32.jpg)
© 2018 Rock Solid Knowledge 32
Summary
• Messaging/Queues powerful way to connect (micro)services that are not temporally bound
• Load balancing / leveling simple to achieve
• RabbitMQ popular messaging platform
• Very flexible
• Designed for high performance
• May need to build architecture for some features