Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture...
Transcript of Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture...
![Page 1: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/1.jpg)
Daniel Meyer | Camunda
Super Fast Workflow
![Page 2: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/2.jpg)
![Page 3: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/3.jpg)
Request Response
SystemClient
Request
Response
Do some calculation
Client waits for the response
![Page 4: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/4.jpg)
Async Work
SystemClient
Request
Ack
Do some calculationlater
Callback
![Page 5: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/5.jpg)
![Page 6: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/6.jpg)
Overview
• My goal• The current Workflow Engine Architecture• The next Generation Workflow Engine
Architecture
![Page 7: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/7.jpg)
My Goal
• Build a better Workflow Engine
• > 100x better throughput• Horizontally Scalable• Easy to use• Updates without downtime• Data Replication• Simple, Composable Architecture
![Page 8: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/8.jpg)
What we have today
• In the open source space: embeddable Java Workflow Engines Activiti Bonita Camunda JBPM ...
![Page 9: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/9.jpg)
Embeddable WF Engines
• Added as a library to an application (used in-process)
• Can also be used remotely via REST but were not designed for this
![Page 10: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/10.jpg)
Embeddable WF Engines
• Design Choices
Shared Database for Persistence Synchronous / blocking Execution Multi-Threaded Execution
![Page 11: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/11.jpg)
Embeddable WF Engines
• Immediate Consequences (Camunda)
Limited Scalability Optimistic Locking at synchronization points
Still Deadlocks (although very low probability)
![Page 12: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/12.jpg)
Embeddable WF Engines
• Other Consequences (Camunda)
Hard to update without downtime Hard use as a platform Different Api & Usage model for Remote vs. Local
> we can to do better
![Page 13: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/13.jpg)
•How can we make this •a lot faster?
![Page 14: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/14.jpg)
Design Principles
• Brokered Architecture• Replicated, Append only Logs• Single Writer Principle• Sharding• Recative, composable Architecture
![Page 15: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/15.jpg)
Design Principles
• Brokered Architecture• Replicated, Append only Logs• Single Writer Principle• Sharding• Recative, composable Architecture
![Page 16: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/16.jpg)
New Architecture
Client
Broker
TCP/IP
ReqestBuffer
ResponseBuffer
LogBuffer
Log
Worker
poll()
poll()
write()
write() read()
append()
![Page 17: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/17.jpg)
Async Task Protocol
BrokerClient
1. create_task()
2. poll_and_lock_tasks()
3. complete_task()
![Page 18: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/18.jpg)
Async Task Protocol
BrokerClient
1. create_task()
2. poll_and_lock_tasks()
3. abort_task()
![Page 19: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/19.jpg)
State Machines and Logs
• Task Instance State Machine (simplified):
• NEW → LOCKED → COMPLETED
...1NEW
1LOCKED
1COMPLETED
Append next
![Page 20: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/20.jpg)
Continuations
complete()
ack
Continue ?
![Page 21: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/21.jpg)
State Machines, Logs and Contiunations
...7COMPLETED
...1NEW
1LOCKED
1COMPLETED
AsyncTask Instance Log
...7ACTIVE
Flow Node Instance Log
...
...
![Page 22: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/22.jpg)
Result
Camunda 7 New Prototype
Create Async Taskwith 512 Bytes payload
~720 / second ~310K / second (in a burst of 1M Tasks)
On this Laptop: Samusung SSD 250G, Intel(R) Core(TM) i7-3520M CPU 4 cores
![Page 23: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/23.jpg)
What's next?
• Finish • Replication & Distribution
• We need feedback from potential Users• This is a long shot
![Page 24: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/24.jpg)
Thank You
• The name is: Daniel Meyer
• Leading the Runtime Infrastructure and Applications Team at Camunda.org
• Twitter: @MeyerDan
![Page 25: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/25.jpg)
What about User / Human Tasks?
25
=> Just a predefined “Worker” with own DB and Tasklist UI
Broker
Tasklist UI
complete_task()query_tasks()claim_task()assign_task()...
User Task Service poll_and_lock()
complete()
API
User Task DB
UserTaskTable
![Page 26: Super Fast Workflow - JAX Finance · 2018. 6. 21. · •The current Workflow Engine Architecture •The next Generation Workflow Engine Architecture. My Goal •Build a better Workflow](https://reader034.fdocuments.us/reader034/viewer/2022051902/5ff29d01d5b03e1f1c76bc82/html5/thumbnails/26.jpg)
What about Cockpit and History?
26
=> “Downloads” all topics and stores in a DB, indexed and optimized for Querying
BrokerqueryXX()
History Servicepoll(topic,offset)
API
History DB
Table
offset
Topic“Everything that happened”
0 1Cockpit UI