Mastering Message Queues - nyholm.tech · Let’s talk about queues. Work queue @tobiasnyholm...
76
Mastering Message Queues Tobias Nyholm @tobiasnyholm @tobiasnyholm
Transcript of Mastering Message Queues - nyholm.tech · Let’s talk about queues. Work queue @tobiasnyholm...
Mastering Message QueuesTobias Nyholm
@tobiasnyholm
@tobiasnyholm
@tobiasnyholm
The data structureTop
Bottom
FrontBack
@tobiasnyholm
Why?
@tobiasnyholm
Tobias Nyholm• Full stack unicorn on Happyr.com
• Certified Symfony developer
• Symfony core member
• PHP-Stockholm
• Open source
@tobiasnyholm
Open source
PHP-cache
HTTPlugMailgun
LinkedIn API clientSwap
Stampie
BazingaGeocoderBundlePHP-Geocoder
FriendsOfApi/boilerplateGuzzle Buzz
CacheBundlePSR7
SymfonyBundleTest
NSA
SimpleBus integrations
PSR HTTP clients
Neo4j
KNP Github API
PHP-Translation
Puli
Assert
Backup-manager/symfony
php-http/httplug-bundle php-http/multipart-stream
php-http/discovery
happyr/normal-distribution-bundle
nyholm/effective-interest-rate
MailgunBundle
league/geotools
@tobiasnyholm
2013
@tobiasnyholm
@tobiasnyholm
@tobiasnyholm
<?php
namespace Webfish\CompanyBundle\Controller;
use Money\Money; use Rawls\BaseBundle\Controller\BaseController; use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route; use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template; use Symfony\Component\HttpFoundation\RedirectResponse; use Symfony\Component\HttpFoundation\Request; use Webfish\CompanyBundle\Entity\Company; use Webfish\CompanyBundle\Entity\SaleImport; use Webfish\CompanyBundle\Form\SalesImportType; use Webfish\CompanyBundle\Model\SalesImportModel;
class ImportController extends BaseController { /** * @param Request $request * * @Route("/import/sales", name="sales_import") * @Template() * * @return array|RedirectResponse */ public function importAction(Request $request) {
“Testing your controllers.”
@tobiasnyholm
@tobiasnyholm
Matthias Noback
@tobiasnyholm
Infrastructure
The world outside
Web browser
TerminalDatabase
Messaging
Filesystem(E)mail
(Matthias Noback)
@tobiasnyholm
Controller Command
Creates
CommandHandler
Command bus
@tobiasnyholm
HTTP Request
Form
Request
Controller
Patient (entity)
Web p
ort
PatientRepository
RegisterPatient- Handler
RegisterPatient (command)
Infras
truc
ture
Applica
tion
Domain
(Matthias Noback)
@tobiasnyholm
Infrastructure
The world outside
Web browser
TerminalDatabase
Messaging
Filesystem(E)mail
(Matthias Noback)
@tobiasnyholm
Hexagonal architecture
https://www.youtube.com/watch?v=fgQWnglnGeU https://matthiasnoback.nl/2015/01/a-wave-of-command-buses/https://martinfowler.com/bliki/CQRS.htmlhttps://hexagonal-architecture.eu/
@tobiasnyholm
What did we win?
@tobiasnyholm
Middlewares!
@tobiasnyholm
Controller Command
Creates
CommandHandler
Command bus
@tobiasnyholm
Controller Command
Creates
CommandHandler
Command bus
Queue
@tobiasnyholm
Let’s talk about queues
@tobiasnyholmWork queue
Producer
Consumer A
Consumer B
@tobiasnyholm
Fanout
Producer
Consumer A
Consumer B
Binding
Binding
Publish/Subscribe
@tobiasnyholm
Direct
Producer
Consumer A
Consumer B
Foo
Baz
Bar
Routing
@tobiasnyholmRouting
Direct
Producer
Consumer A
Consumer B
Foo
Foo
@tobiasnyholmTopics
Topic
Producer
Consumer A
Consumer B
*.error
app.*
firewall.#
app.error
firewall.noticeapp.notice
@tobiasnyholm
Let’s see some code
@tobiasnyholm
$ composer req symfony/messenger
@tobiasnyholm
Controller Message
Creates
Message handler
Message bus
@tobiasnyholm<?php
namespace App\Message\Command;
class SendNotification { private $message; private $users;
public function __construct(string $message, array $users) { $this->message = $message; $this->users = $users; }
public function getMessage(): string { return $this->message; }
public function getUsers(): array { return $this->users; } }
@tobiasnyholm<?php
namespace App\Controller;
use App\Message\Command\SendNotification; use Symfony\Component\Messenger\MessageBusInterface; // ...
class DefaultController { public function index(MessageBusInterface $bus, Request $request) { $users = ['samuel', 'christelle']; $message = $request->query->get('message', 'Something.');
$bus->dispatch(new SendNotification($message, $users));
return new Response('<html><body>OK.</body></html>'); } }
@tobiasnyholm
@tobiasnyholm
<?php
namespace App\MessageHandler;
use App\Message\Command\SendNotification;
class SendNotificationHandler { public function __invoke(SendNotification $message) { foreach ($message->getUsers() as $user) { echo "Send notification to... ".$user."\n"; } } }
@tobiasnyholm
# config/services.yaml services: App\Message\CommandHandler\SendNotificationHandler: tags: - messenger.message_handler
# — — — — — —
# config/services.yaml services: App\Message\CommandHandler\: resource: ‘../src/Message/CommandHandler/*’ tags: ['messenger.message_handler']
@tobiasnyholm
@tobiasnyholm
Message Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Transports
@tobiasnyholm
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Message
Transports
• Sender
• Receiver
• Configurable with DSN
• Encode/Decode
@tobiasnyholm
# config/packages/messenger.yaml framework: messenger: transports: default: '%env(MESSENGER_DSN)%'
routing: ‘App\Message\Command\SendNotification’: default
# -------------
# .env MESSENGER_DSN=amqp://guest:guest@localhost:5672/%2f/messages
@tobiasnyholm
@tobiasnyholm
$ bin/console messenger:consume-messages
@tobiasnyholm
@tobiasnyholm
@tobiasnyholm
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Message
@tobiasnyholm
@tobiasnyholm
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Message
@tobiasnyholm
Transport
dispatch
Transport
send
Workerreceive
Message handlerhandles
Message handler
dispatch
App 1
TransportTransport Workerreceive
Message handler handles
Message handler
dispatch
App 2
MessageMessage
@tobiasnyholm
# config/packages/messenger.yaml framework: messenger: transports: default: ‘%env(MESSENGER_DSN)%' enqueue: '%env(ENQUEUE_DSN)%'
routing: ‘App\Message\Command\SendNotification': [default, enqueue]
# — — — — — —
# .env MESSENGER_DSN=amqp://guest:guest@localhost:5672/%2f/messages ENQUEUE_DSN=enqueue://acme
@tobiasnyholm
Direct
Producer
Consumer A
Consumer B
Foo
Baz
Bar
TransportTransport Workerreceive
Message handler handles
Message handler
dispatch
App 2
@tobiasnyholm
Multiple busses
@tobiasnyholm
Multiple busses
Command Command Handler
Event
Query Query Handler
Event SubscriberEvent SubscriberEvent Subscriber
@tobiasnyholm
Multiple busses# config/packages/messenger.yaml framework: messenger: # The bus that is going to be injected when injecting MessageBusInterface: default_bus: messenger.bus.commands
# Create buses buses: messenger.bus.command: ~ messenger.bus.query: ~ messenger.bus.event: ~
@tobiasnyholm
Multiple busses# config/packages/messenger.yaml framework: messenger: # The bus that is going to be injected when injecting MessageBusInterface: default_bus: messenger.bus.commands
# Create buses buses: messenger.bus.command: middleware: - messenger.middleware.exactly_one_handler - messenger.middleware.validation - 'App\Middleware\EventStoreMiddleware' messenger.bus.query: middleware: - messenger.middleware.exactly_one_handler - messenger.middleware.validation messenger.bus.event: middleware: - messenger.middleware.allow_no_handler - messenger.middleware.validation
@tobiasnyholm
Multiple bussesservices: _defaults: # ...
bind: $commandBus: '@messenger.bus.command' $queryBus: '@messenger.bus.query' $eventBus: '@messenger.bus.event'
@tobiasnyholm<?php
namespace App\Message\QuaryHandler;
use App\Message\Quary\GetUserForNotification;
class GetUsersForNotificationHandler { public function __invoke(GetUserForNotification $query) { $users = $this->getUsers($query->getSubject());
return $users; }
private function getUsers(string $subject) { // TODO Decide upon $subject return ['tobias', ‘maria', 'samuel']; } }
@tobiasnyholm
Send events from handlerclass SendNotificationHandler { private $eventBus;
public function __construct(MessageBusInterface $eventBus) { $this->eventBus = $eventBus; }
public function __invoke(SendNotification $message) { foreach ($message->getUsers() as $user) { echo "Send notification to... ".$user."\n"; }
$this->eventBus->dispatch(new NotificationSent()); } }
@tobiasnyholm
Handle failure
@tobiasnyholm
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Requeue
Message
@tobiasnyholm
Error
@tobiasnyholm
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Requeue
Message
@tobiasnyholm
Create new entities
@tobiasnyholm
<?php
namespace App\Controller;
use App\Message\SendNotification; use Symfony\Component\Messenger\MessageBusInterface; // ...
class UserController { public function create(MessageBusInterface $bus, Request $request) { $bus->dispatch(new CreateUser($request->request->get('username', '')));
return new Response('See /api/user/{user_id}’); } }
@tobiasnyholmuse Symfony\Component\Validator\Constraints as Assert;
class CreateUser { /** * @Assert\Uuid() * @Assert\NotBlank() */ private $uuid; /** * @Assert\NotBlank() * @Assert\Length(min=3) */ private $username;
public function __construct(string $uuid, string $username) { $this->uuid = $uuid; $this->username = $username; }
public function getUuid(): string { return $this->uuid; }
public function getUsername(): string { return $this->username;
@tobiasnyholm<?php
namespace App\Controller;
use App\Message\SendNotification; use Symfony\Component\Messenger\MessageBusInterface; // ...
class UserController { public function create(MessageBusInterface $bus, Request $request) { $uuid = (string) Uuid::uuid4(); $bus->dispatch(new CreateUser($uuid, $request->request->get('username', '')));
return new Response('See /api/user/'.$uuid); } }
@tobiasnyholmuse Symfony\Component\Validator\Constraints as Assert;
class CreateUser { /** * @Assert\Uuid() * @Assert\NotBlank() */ private $uuid; /** * @Assert\NotBlank() * @Assert\Length(min=3) */ private $username;
public function __construct(string $uuid, string $username) { $this->uuid = $uuid; $this->username = $username; }
public function getUuid(): string { return $this->uuid; }
public function getUsername(): string { return $this->username;
@tobiasnyholm
Cache everything
HTTP RequestThin app
Cache
TranspoTransport Workerreceive
Message handlesMessage handler
dispatch
GET
POST
@tobiasnyholm
Handle 1M requests / second
Thin app
Workers Workers
Workers
HTTP Request
@tobiasnyholm
Handle 1M requests / second
AWS Lambda
Workers Workers
Workers
HTTP Request
@tobiasnyholm
Questions?
@tobiasnyholm
Envelope
Message handler
Message bus
Message handlerdispatch handles
Worker
dispatc
h
TransportTransport
receive
send
Message
@tobiasnyholm
Questions?@tobiasnyholm
