Post on 15-Apr-2017
Matt Walters @mateodelnorte hi@iammattwalters.com
How to CQRS in Node
Eventually consistent architectures that scale and grow.
It’s real. It works!
Former TechStars Co. CTO. Now consultant.
Built two businesses’ platforms from scratch using CQRS in Node. Both large, distributed systems.
CQRS made both more maintainable and extendable.
Marketing platform crunched the Twitter firehose in realtime. 3 Engineers managed around 20 services. - GoChime.com
Bond Exchange with over $1.75B in trades. 6 engineers managed around 40 services. - Electronifie.com
Open Source my tooling and frameworks.
Other companies using them too!
(and I’ll tell you how)
That’s me ^^ !!
When to CQRS• Realtime, reactive systems • When preferring small, modular services • When aiming for learning and growth • When aiming to grow or eventually split teams • Want ability to scale different parts of your
system separately
When not to CQRS
• Standalone, static sites • Standalone, simple CRUD applications
First, a primer from history.
Bertrand Meyer, regarding object interfaces:
“Every method should either be a command that performs an action, or a query that returns data to the caller, but not both. In other words, Asking a question should not change the answer.”
Long before CQRS was CQS:
Command Query Separation.
this guy
Some Type
doSomething() : void
getSomeStuff() : Stuff
Either change stuff<——
Or get stuff<——
CQSCommand-Query
Separation
doAndGetStuff() : Stuff Never both!——————
Command-Query Responsibility Segregation
A system-wide architecture that states - externally facing subsystems (apps and apis) send commands to perform actions which update the system’s state and request queries to determine system’s state.
*Basically CQS on a system-wide scale. Calls between services should change stuff, or get stuff. Never both.
CQRS
Command-Query Responsibility Segregation
CQRS also denotes that queries and command processing are provided by different subsystems.
Queries are made against a data store. Commands are sent to and processed by services.
CQRSOne more thing!
What is CQRS?
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-appweb client
denormalizer
web-uiweb-uisvc
unidirectional floweventually consistent
Queries
Commands
the dance!Events
How about a larger system?
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
Queries
Commands
unidirectional floweventually consistent
the dance!Events
What’s that dance you’re doing?
denormalizer dbdenormalizer dbdenormalizer db
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
the dance!
chain reaction of events which play out as a result of an incoming command.
each service subscribes to the events they care about
choreography!(not orchestration)
Events
commands tell services when an actor wants an action
clients send commands to instruct a service to do work
commands are sent asynchronously; fire and forget
commands are present-tense, directives: order.create
web app order-svc
order.create
commands are sent directly to a single receiving service
events tell the world when you’re done
services publish to inform other services of work / actions performed, and state updated
services publish (broadcast) events to any services that wish to subscribe
events past-tense, describe what happened: order.created
order-svc fulfillment-svc
order.createdorder.createdorder.created
order.created
Two types of services
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
front end
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
back end
Two types of services
web-uiweb-uiweb-appweb client
denormalizer
web-uiweb-uisvc
front end
denormalizer dbdenormalizer dbdenormalizer db
What’s different?
Let’s focus on:
web-uiweb-uiweb-appweb client
web-uiweb-uisvc
front end (an app’s perspective)
denormalizer dbdenormalizer dbdenormalizer db
What’s different?
Apps (and apis) still query a db to get the state of the system
Never directly modify the db they read from
Let’s focus on:
web-uiweb-uiweb-appweb client
web-uiweb-uisvc
denormalizer dbdenormalizer dbdenormalizer db
What’s different?
Instead, apps (and apis) send commands instructing services to perform an action
Apps expect their read only representation of system state will eventually be updated in the denormalizer db
Let’s focus on:front end (an app’s perspective)
Commands are sent over a reliable transport (rabbitmq, kafka, zeromq, etc) to ensure delivery and eventual consistency
rabbitmqmessaging that just works
• direct send to queue • fanout / topic routing • highly available • highly performant • used in financial exchanges, industrial applications and more • open source • free
servicebussuper simple messaging in node
• direct send • pub / sub / fanout / topic -routing • simple to set up • highly performant • used in financial exchanges, online advertising and more • open source • free • perfect for creating microservices!
web-uiweb-uiweb-appweb client
denormalizer
web-uiweb-uisvc
front end
denormalizer dbdenormalizer dbdenormalizer db
What’s different?
Let’s focus on:
web-uiweb-uiweb-app
web-uiweb-uisvc
front end commands
denormalizer dbdenormalizer dbdenormalizer db
How’s this work?
Let’s focus on:
Sending commands from the front end
// web-app, onButtonClick. instead of updating db. const bus = require('servicebus').bus();
bus.send(‘order.create', { order: {
userId: userId, orderItems: items
} });
web-uiweb-uiweb-appweb-uiweb-uisvc
// fire and forget.
command name
command itself
order.create command
Then what from the front end?
web-uiweb-uiweb-appdenormalizer
dbdenormalizer dbdenormalizer db
We wait.
• reactive / realtime: • mongo oplog tailing (meteor) • rethinkdb • redis notifications • couchdb • graphQL • polling
• non-realtime: • product design: thanks! we’re processing your
order! check back later for updates!
queries!
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
front end
Let’s focus on:
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
back end
Let’s focus on:
back end
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
Let’s focus on:
back end (a service’s perspective)
web-uiweb-uiweb-app
svc
Commands
Events
• Listen for commands and subscribe to events • Performs business logic to process commands and events • Update local state (optionally) • Publish events to tell external services of updated state
Let’s focus on:
Sample service.
web-uiweb-uiweb-appweb-uiweb-uisvc
command name
command object
// order-svc index.js const bus = require(‘./bus’); const create = require(‘./lib/create’);
bus.listen(‘order.create', (event) => { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
service publishes to the world when it’s done!
order.create command order.created
event
Sample service.
web-uiweb-uiweb-appweb-uiweb-uisvc
listening for commandsperforming business logic & updating state// order-svc index.js
const bus = require(‘./bus’); const create = require(‘./lib/create’);
bus.listen(‘order.create', (event) => { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
order.create command order.created
event
completing atomic transaction and allowing error handling
back end (a downstream service’s perspective)
web-uiweb-uiweb-app
svc
Commands
Events
• Listen for commands and subscribe to events • Performs business logic to process commands and events • Update local state (optionally) • Publish events to tell external services of updated state
svc-2
Even
ts
Events
Let’s focus on:
Same thing!
Sample downstream service.
web-uiweb-uiweb-appweb-uiweb-uisvc
// fulfillment-svc index.js const bus = require(‘./bus’); const fulfill = require(‘./lib/fulfill’);
bus.subscribe(‘order.created', (event) => { fulfill(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.fulfilled’, order, () => { event.handle.ack();
}); }); });
order.created order.fulfilled
subscribe for events instead of listening for commands
no different, from any other service!
servicebus-register-handlersconvention based event handler definition for
distributed services using servicebus.
automatically registers event & command handlers saved as modules in folder
initialize at startup
servicebus-register-handlers
const bus = require(‘./lib/bus'); // instantiate servicebus instance const config = require('cconfig')(); const log = require('llog'); const registerHandlers = require('servicebus-register-handlers');
registerHandlers({ bus: bus, handleError: function handleError (msg, err) {
log.error('error handling %s: %s. rejecting message w/ cid %s and correlationId %s.', msg.type, err, msg.cid, this.correlationId);
log.error(err);
msg.handle.reject(function () { throw err; });
}, path: './lib/handlers', queuePrefix: 'my-svc-name' });
initialize at startup:
provide initialized busdefine your error handling
path to your handlers
prefix to differentiate similar queues
servicebus-register-handlers
const log = require("llog");
module.exports.ack = true;
module.exports.queueName = 'my-service-name-order'; module.exports.routingKey = "order.create"; module.exports.listen = function (event, cb) {
log.info(`handling listened event of type ${event.type} with routingKey ${this.routingKey}`);
/*
do something with your event
*/
cb(); };
each handler is a file
no params marks success. pass back error to retry or fail.
callback based transactions!
differentiate queues for different services
specify which commands or events to listen or subscribe to
servicebus-register-handlerssuper simple messaging in node
npm install servicebus-register-handlers —save
What about the ‘work’ part?
const log = require("llog");
module.exports.ack = true;
module.exports.queueName = 'my-service-name-order'; module.exports.routingKey = "order.create"; module.exports.listen = function (event, cb) {
log.info(`handling listened event of type ${event.type} with routingKey ${this.routingKey}`);
/*
do something with your event
*/
cb(); };
// order-svc index.js const bus = require(‘./bus’); const create = require(‘./lib/create’);
bus.listen(‘order.create', (event) => { create(event, (err, order) => { if (err) return event.handle.reject(); bus.publish(‘order.created’, order, () => { event.handle.ack();
}); }); });
these parts
What about the ‘work’ part?That’s up to you!
Need an audit trail? Targeting finance? Consider event sourcing. *and my framework, ‘sourced’
Depending on your problem, the right choice could be mongoose and mongodb, a graph database, an in-memory data structure, or even flat files.
CQRS makes no assertions about what technology you should use, and in fact frees you to make a different decision for each particular problem.
and depends on the problem you’re solving
But wait! There’s more!servicebus middleware!
middleware can inspect and modify incoming and outgoing messages
// ./lib/bus.js required as single bus instance used anywhere in service const config = require('cconfig')(); const servicebus = require('servicebus'); const retry = require('servicebus-retry');
const bus = servicebus.bus({ url: config.RABBITMQ_URL });
bus.use(bus.package()); bus.use(bus.correlate()); bus.use(retry({ store: new retry.RedisStore({ host: config.REDIS.HOST, port: config.REDIS.PORT }) }));
module.exports = bus;
bus.use() middleware into bus message pipeline. middleware can act on incoming and/or outgoing messages
But wait! There’s more!servicebus middleware!
middleware can inspect and modify incoming and outgoing messages
// ./lib/bus.js required as single bus instance used anywhere in service const config = require('cconfig')(); const servicebus = require('servicebus'); const retry = require('servicebus-retry');
const bus = servicebus.bus({ url: config.RABBITMQ_URL });
bus.use(bus.package()); bus.use(bus.correlate()); bus.use(retry({ store: new retry.RedisStore({ host: config.REDIS.HOST, port: config.REDIS.PORT }) }));
module.exports = bus;
packages outgoing message data and adds useful type, timestamp, and other properties
But wait! There’s more!servicebus middleware!
middleware can inspect and modify incoming and outgoing messages
// ./lib/bus.js required as single bus instance used anywhere in service const config = require('cconfig')(); const servicebus = require('servicebus'); const retry = require('servicebus-retry');
const bus = servicebus.bus({ url: config.RABBITMQ_URL });
bus.use(bus.package()); bus.use(bus.correlate()); bus.use(retry({ store: new retry.RedisStore({ host: config.REDIS.HOST, port: config.REDIS.PORT }) }));
module.exports = bus;
adds a correlationId for tracing related commands and events through your system
But wait! There’s more!servicebus middleware!
middleware can inspect and modify incoming and outgoing messages
// ./lib/bus.js required as single bus instance used anywhere in service const config = require('cconfig')(); const servicebus = require('servicebus'); const retry = require('servicebus-retry');
const bus = servicebus.bus({ url: config.RABBITMQ_URL });
bus.use(bus.package()); bus.use(bus.correlate()); bus.use(retry({ store: new retry.RedisStore({ host: config.REDIS.HOST, port: config.REDIS.PORT }) }));
module.exports = bus;
now, every failed message will retry 3 times if errors occur. after that, the message will automatically be put on an error queue for human inspection!
And more!distributed tracing middleware!
var trace = require('servicebus-trace'); bus.use(trace({ serviceName: 'my-service-name', store: new trace.RedisStore({ host: config.REDIS_HOST || 'localhost', port: config.REDIS_PORT || 6379 }) }));
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
back end
Recapping Back End Services
back end services
web-uiweb-uiweb-app
svc
Commands
Events
• Listen for commands and subscribe to events • Performs business logic to process commands and events • Update local state (optionally) • Publish events to tell external services of updated state
Recapping:
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
back end
Recapping Back End Services
wat wat watwat
back end
What’s a denormalizer?
front end
• Just another back end service • Has one job to do • Subscribe to all events that the UI cares about • Persist events in a format most efficient for the UI to view• Completes the eventually consistent, unidirectional flow
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web client
denormalizer
Eventssvc
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web client
denormalizer
back end
What’s a denormalizer?
Events
front end
svc
Recapping the big picture.
denormalizer dbdenormalizer dbdenormalizer db
web-uiweb-uiweb-app
web-uiweb-uiweb-api
web client
mobile client
denormalizer
svc-3
svc-2web-uiweb-uisvc-1
Queries
Commands
unidirectional floweventually consistent
the dance!Events
Matt Walters github & twitter: @mateodelnorte email: hi@iammattwalters.com website: iammattwalters.com
rabbitmq.com
npmjs.com/package/servicebus
How to CQRS in Node
npmjs.com/package/servicebus-register-handlersnpmjs.com/package/servicebus-retrynpmjs.com/package/servicebus-trace