Lost in transaction? - JAX London...If your transaction involves 2 to 4 steps, choreography might be...
79
@berndruecker Lost in transaction? Strategies to manage consistency in distributed systems
Transcript of Lost in transaction? - JAX London...If your transaction involves 2 to 4 steps, choreography might be...
@berndruecker
Lost in transaction?
Strategies to manage consistencyin distributed systems
@berndruecker
Order
OrderLine Item
Customer
Customer
Address
orderIdcustomerIdchargeId…
customerId…
productIdprice…
Charge
Charge
Order
chargeIdamountpaymentProvidertransactionId…
Aggregate
Entity
Value Object
Reference by identity@berndruecker
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice… Order
Charge
Charge
chargeIdamountpaymentProvidertransactionId… Create
Charge
Mark order as
paid
All ornothing+
@berndruecker
ACID
AtomicityConsistencyIsolationDurability
@berndruecker
Aggregates = Consistency Boundaries
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice… Order
Charge
Charge
chargeIdamountpaymentProvidertransactionId…
You can do ACID within here
Or here
But not a joinedACID transaction!
@berndruecker
Aggregates = Consistency Boundaries
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice… Order
Charge
Charge
chargeIdamountpaymentProvidertransactionId…
Own DB / autonomous technology decisionsAPI-drivenScalability…
@berndruecker
Aggregates = Consistency Boundaries
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice… Order
Charge
Charge
chargeIdamountpaymentProvidertransactionId…
Own DB / autonomous technology decisionsAPI-drivenScalability…
@berndruecker
Boundaries need to be designed carefully
Charge
chargeIdamountpaymentProvidertransactionId…
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice…Order
@berndruecker
But no implicit constraints!
Order
OrderLine Item
orderIdcustomerIdchargeId…
productIdprice… Order
Charge
Charge
chargeIdamountpaymentProvidertransactionId…
Joined DB transaciton
@berndruecker
There is two-phase commit (XA)!!
TXCoordinator
ResourceManagers
PreparePhase
CommitPhase
@berndruecker
Pat Helland
“
Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
@berndruecker
Pat Helland
Grown-Ups Don’t Use Distributed Transactions
“
Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
@berndruecker
Starbucks does not use two phase commithttps://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.html
Photo by John Ingle
@berndruecker
Eric BrewerAtomicityConsistencyIsolationDurability
http://pld.cs.luc.edu/courses/353/spr11/notes/brewer_keynote.pdf @berndruecker
That means
Do A
Do B
Temporarilyinconsistent
Eventuallyconsistentagain
t
Consistent
LocalACID
LocalACID
1 aggregate1 (micro-)service1 program1 resource
Violates „I“ of ACID
@berndruecker
„Building on Quicksand“ Paper
ACID2.0Pat Helland
@berndruecker
AssociativeCommutativeIdempotentDistributed2.0
(a + b) + c = a + (b + c)
a + b = b + a
f(x) = f( f(x) )
„Building on Quicksand“ Paper
Pat Helland
@berndruecker
Distributed@berndruecker
Distributed systems @berndruecker
Network problems
CreditCard
Paymentcharge
@berndruecker
Strategy: retry
CreditCard
Payment
Charge Credit CardcardNumber
amount
Charge Credit CardcardNumber
amounttransactionId
Not idempotent
Idempotent
has to be idempotent
charge
@berndruecker
Photo by pixabay, available under Creative Commons CC0 1.0 license.
@berndruecker
Requirement: Idempotency of services!
Photo by pixabay, available under Creative Commons CC0 1.0 license.
@berndruecker
Requirement: Idempotency of services!
Photo by Chr.Späth, available under Public Domain.
@berndruecker
Distributed systems @berndruecker
It is impossible todifferentiate certain
failure scenarios:
Independant ofcommunication style!
Service Provider
Client
@berndruecker
Strategy: Cleanup
CreditCard
Paymentcharge
Make sure it isnot charged!
Cancel chargecardNumber
amounttransactionId
Raisepayment failed
@berndruecker
Distributed systems @berndruecker
Some communication challengesrequire state.
@berndruecker
Strategy: Stateful retry
CreditCard
Paymentcharge
@berndruecker
Strategy: Stateful retry
CreditCard
Paymentcharge
Make sure it isnot charged!
@berndruecker
Warning:Contains Opinion
Payment
Stateful retry
CreditCardREST
@berndruecker
Stateful retry & cleanup
CreditCard
PaymentREST
Cancelcharge
@berndruecker
Architecture
Infrastructure
Application
Domain
WorkflowEngine
@berndruecker
Architecture
Infrastructure
Application
Domain
WorkflowEngine
@berndruecker
Lightweight Open SourceBPMN
Live hacking
https://github.com/flowing/flowing-retail/tree/master/rest
@berndruecker
Embedded Engine Example (Java)
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
@berndruecker
A relatively common pattern
Service(e.g. Go)
Rabbit
RDMS
1. Receive
4. Send additional events
2. Business Logic
3. Send response? ACK
@berndruecker
„Can this handle 15k requests per second?“
@berndruecker
and
ACID?
@berndruecker
@berndruecker
Compensation – the classical example
Sagabookhotel
bookcar
bookflight
cancelhotel
cancelcar
1. 2. 3.
5.6.
In case of failuretrigger compensations
booktrip
2 alterntive approaches: choreography & orchestration
@berndruecker
Event-driven choreography
Hotel
Flight
Car
Trip
Tripbooked
Flightbooked
Triprequested
Hotelbooked
Carbooked
Requesttrip
@berndruecker
Event-driven choreography
Hotel
Flight
Car
Trip
Tripfailed
Triprequested
Hotelbooked
Carbooked
Requesttrip
Flightfailed
Car canceled
Hotel canceled
Perform undo(cancel car booking)
Perform undo(cancel hotel)
@berndruecker
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker
If your transaction involves 2 to 4 steps, choreography might be a very good fit.
However, this approach can rapidly become confusing if you keep adding extra steps in your transaction as it is difficult to track which services listen to which events. Moreover, it also might add a cyclic dependency between services as they have to subscribe to one another’s events.
Denis RosaCouchbase
https://blog.couchbase.com/saga-pattern-implement-business-transactions-using-microservices-part/
@berndruecker
Implementing changes in the process
Hotel
Flight
Car
Trip
Tripfailed
Triprequested
Hotelbooked
Carbooked
Requesttrip
Flightfailed
Car canceled
Hotel canceled
We have a new basic agreementwith the car rental agency and can cancel for free within 1 hour
– do that first!
@berndruecker
Implementing changes in the process
Hotel
Flight
Car
Trip
Tripfailed
Triprequested
Hotelbooked
Carbooked
Requesttrip
Flightfailed
Car canceled
Hotel canceled
You have to adjust all services and redeploy at the same time!
We have a new basic agreementwith the car rental agency and can cancel for free within 1 hour
– do that first!
@berndruecker
What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
@berndruecker
Orchestration
Hotel
Flight
Car
Trip
Tripbooked
Requesttrip
Bookhotel
Hotelbooked
Carbooked
Flightbooked
Book car
Bookflight
@berndruecker
Orchestration
Hotel
Flight
Car
Trip
Tripbooked
Requesttrip
Bookhotel
Hotelbooked
Carbooked
Flightbooked
Book car
Bookflight
We have a new basic agreementwith the car rental agency and can cancel for free within 1 hour
– do that first!
You have to adjust one service and redeploy only this one!
@berndruecker
Describe orchestration with BPMN
Trip
Tripbooked
Requesttrip
@berndruecker
The workflow is domain logic as part of the service
Trip
@berndruecker
The workflow is domain logic as part of the service
Trip
Payment
Payment could be onestep in the Trip Saga
@berndruecker
Caitie McCaffrey | @caitie
@berndruecker
Graphical models?
@berndruecker
Clemens VastersArchitect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
Clemens VastersArchitect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
Clemens VastersArchitect at Microsoft
http://vasters.com/archive/Sagas.html
@berndruecker
BPMNBusiness Process
Model and Notation
ISO Standard
@berndruecker
Living documentation for long-running behaviour
@berndruecker
Visual HTML reports for test cases
@berndruecker
BizDevOps
@berndruecker
Fancy a DSL? Just do it!
https://github.com/berndruecker/flowing-trip-booking-saga
The visual getauto-generated…
@berndruecker
Thoughts on the state machine | workflow engine market
@berndruecker
Thoughts on the state machine | workflow engine market
OSS Workflow orOrchestration Engines
Stack Vendors,Pure Play BPMS
Low Code Platforms
Homegrown frameworksto scratch an itch
Integration Frameworks
Cloud Offerings
Uber, Netflix, AirBnb, ING, … AWS Step Functions, Azure Durable Functions, …
Camunda, Zeebe, jBPM, Activiti, Mistral, …
PEGA, IBM, SAG, …
Apache Airflow, Spring Data Flow, …
Apache Camel, Balerina, …
Data Pipelines
@berndruecker
Does it support stateful operations?
Does it support the necessary flow logic?
Does it support BizDevOps?
Does it scale?
@berndruecker
My personal pro-tip for a shortlist ;-)
OSS Workflow orOrchestration Engines
Stack Vendors,Pure Play BPMS
Low Code Platforms
Homegrown frameworksto scratch an itch
Integration Frameworks
Cloud Offerings
Data Pipelines
Camunda & Zeebe
@berndruecker
Recap
• Aggregates = Consistency boundaries• Grown ups don‘t use distributed transactions
but eventual consistency• Idempotency is super important• Some consistency challenges require state• Stateful retry & cleanup• Saga / Compensation
@berndruecker
Thank you!
@berndruecker
[email protected]@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/application-development/3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/
![Loading [MathJax]/jax/output/CommonHTML/jax](https://static.fdocuments.us/doc/165x107/625335b291158e3f663c6c05/loading-mathjaxjaxoutputcommonhtmljax.jpg)
