Complex event flows indistributed systems

@berndruecker With thoughts from http://flowing.io@berndruecker | @martinschimak

3 common hypotheses I check today:

# Events decrease coupling# Orchestration needs to be avoided# Workflow engines are painful

Berlin, Germany

bernd.ruecker@camunda.com@berndruecker

Bernd RueckerCo-founder and Developer Advocate ofCamunda

Simplified example:dash button

Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/

Three steps…

Who is involved? Some bounded contexts…

Checkout

Payment

Inventory

Shipment

(Micro-)services

Checkout

Payment

Inventory

Shipment

Autonomous (micro-)services

Checkout

Payment

Inventory

Shipment

Dedicated Application Processes

Dedicated infrastructure

Dedicated Development Teams

Events decrease coupling

Example

Checkout

Payment

Inventory

Shipment

The button blinks if we canship within 24 hours

Request/response: temporal coupling

Checkout

Payment

Inventory

Shipment

RequestResponse

The button blinks if we canship within 24 hours

Temporal decoupling with events and read models

Checkout

Payment

Inventory

Shipment

GoodStored

Read Model

GoodFetched

The button blinks if we canship within 24 hours

*Events are facts about what happened (in the past)

Events can decrease coupling*

*e.g. decentral data-management, read models, extract cross-cutting aspects

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Order placed

Payment received

Goodsshipped

Goodsfetched

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Order placed

Payment received

Goodsshipped

Goodsfetched

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Order placed

Payment received

Goodsshipped

Goodsfetched

Fetch the goodsbefore thepayment

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Order placed

Payment received

Goodsshipped

Goodsfetched

Fetch the goodsbefore thepayment

Customers canpay via invoice

…

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Order placed

Payment received

Goodsshipped

Goodsfetched

Fetch the goodsbefore thepayment

Peer-to-peer event chains

Checkout

Payment

Inventory

Shipment

Fetch the goodsbefore thepayment

Goodsfetched

Order placed

Payment received

Goodsshipped

Photo by born1945, available under Creative Commons BY 2.0 license.

Order

Extract the end-to-end responsibility

Checkout

Payment

Inventory

Shipment

*Commands have an intent about what needs to happen in the future

Order placed

Paymentreceived

Retrievepayment

Commands help to avoid (complex)peer-to-peer event chains

Orchestration needs to be avoided

Smart ESB-like middleware

Checkout

Payment

Inventory

Shipment

Order

Order placed

Payment received

Goodfetched

Goodshipped

Dumb pipes

Checkout

Payment

Inventory

Shipment

Order

Smart endpoints and dumb pipes

Martin Fowler

Danger of god services?

Checkout

Order

A fewsmart god services tellanemic CRUD services what to do

Sam Newmann

Payment

Inventory

Shipment

Danger of god services?

Checkout

Payment

Inventory

Shipment

Order

A fewsmart god services tellanemic CRUD services what to do

Sam Newmann

A god service is only createdby bad API design!

Example

Order Payment

RetrievePayment

Example

Order PaymentCreditCard

RetrievePayment

Example

Order PaymentCreditCard

RetrievePayment

Rejected

Example

Order Payment

Client of dumb endpoints easily become a god services.

If the creditcard was

rejected, thecustomer canprovide new

details

CreditCard

RetrievePayment

RejectedRejected

Paymentfailed

Who is responsible to deal with problems?

Order Payment

If the credit card was

rejected, the customer can provide new

details

CreditCard

RetrievePayment

Rejected

Paymentreceived

Paymentfailed

Long-running execution

Order Payment

Clients of smart endpoints remains lean.

If the credit card was

rejected, the customer can provide new

details

CreditCard

RetrievePayment

Rejected

Paymentreceived

Smart endpoints are potentially long-running

Persist thing(Entity, Document, Actor, …)

State machine orworkflow engine

Typicalconcerns

DIY = effort, accidentalcomplexity

Scheduling, Versioning, operating, visibility, scalability, …

Handling State

Workflow engines are painful

Complex, proprietary, heavyweight, central, developer adverse, …

Avoid the wrong tools!

Death by properties panel

Low-code is great!(You can get rid

of your developers!)

Complex, proprietary, heavyweight, central, developer adverse, …

Workflow engines, state machines

It is

relevantin modern

architectures

CADENCE

Silicon valleyhas recognized

Workflow engines, state machines

CADENCEWorkflow engines, state machines

CADENCE

also at scaleWorkflow engines, state machines

Zeebe.io

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

What do I mean by„leightweight?“

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

Build enginein one line ofcode(using in-memory H2)

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

Define flowe.g. in Java DSL

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

Define flowe.g. in Java DSL

BPMNBusiness Process

Model and Notation

ISO Standard

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

We can attachcode…

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

…that iscalled whenworkflowinstances pass through

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();

engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()

.done()).deploy();

engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));

}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));

}}

Start instances

Payment

Now you have a state machine!

Payment

Easy to handle time

martin.schimak@plexiti.com@martinschimak

Synchronous communication

RESTOrder Payment

CreditCard

Synchronous communication

RESTOrder Payment

CreditCard

Synchronous communication

REST

Order PaymentCreditCard

Synchronous communication

RESTOrder Payment

CreditCard

StatefulRetry

Distributedsystems

It is impossible todifferentiate certain

failure scenarios.

Independant ofcommunication style!

Service Provider

Client

Distributed systems introduce complexity you have to tackle!

CreditCard

PaymentREST

Distributed systems introduce complexity you have to tackle!

CreditCard

PaymentREST

Distributedsystems

Distributed transactions using compensation *

Compensation

Relaxed consistency

Temporarily inconsistent

state

But eventually consistent

No Isolation (as in ACID)

https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

<= milliseconds seconds minutes, weeks, …

Business

IT

Business processesautomation

Distributed Transactions

Orchestration

Communication in distributed systems

long runningalways short running short running, but potentially long running

Use cases for workflow automation

Biz Devimprove

communicationimprove

communication

Ops

Biz Dev

Leveragestate machine & workflow engine

Living documentation

Visibility in testing

improvecommunication

improvecommunication

Ops

Visual HTML reports for test cases

Biz Dev

Leveragestate machine & workflow engine

Living documentation

Visibility in testing

Understand and discussbusiness processes

Evaluate optimizationsin-sync with

implementation

improvecommunication

improvecommunication

Ops

Living documentation for long-running behaviour

Biz Dev

Leveragestate machine & workflow engine

Living documentation

Visibility in testing

Understand and discussbusiness processes

Evaluate optimizationsin-sync with

implementation

improvecommunication

improvecommunication

Ops

Biz Dev

Leveragestate machine & workflow engine

Living documentation

Visibility in testing

Operate with visibilityand context

Understand and discussbusiness processes

Evaluate optimizationsin-sync with

implementation

improvecommunication

improvecommunication

Ops

ProperOperations

Visibility + Context

Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on.

Jimmy Floyd, 24 Hour Fitnesse

„

Workflows live inside service boundaries

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

…

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

Lightweight workflow engines aregreat* – don‘t DIY

*e.g. enabling potentially long-running services, solving harddeveloper problems, can run decentralized

Sales-Order & Order-Fulfillment

via Camunda

for every order worldwide

(Q2 2017: 22,2 Mio)

Code example & live demo

InventoryPaymentOrder ShippingCheckout Monitor

https://github.com/flowing/flowing-retail/

Human Tasks

H2 H2

# Events decrease coupling: sometimesread-models, but no complex peer-to-peer event chains!

# Orchestration needs to be avoided: sometimesno ESB, smart endpoints/dumb pipes, important capabilities need a home

# Workflow engines are painful: some of themlightweight engines are easy to use and can run decentralized,they solve hard developer problems, don‘t DIY

Thank you!

bernd.ruecker@camunda.com@berndruecker

https://bernd-ruecker.com

https://blog.bernd-ruecker.com

https://github.com/flowing

https://www.infoq.com/articles/events-workflow-automation

With thoughts from http://flowing.io@berndruecker | @martinschimak

Contact:

Slides:

Blog:

Code:

https://www.infoworld.com/article/3254777/application-development/3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html

https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/