Microservices for a Streaming World
-
Upload
ben-stopford -
Category
Software
-
view
8.830 -
download
3
Transcript of Microservices for a Streaming World
![Page 1: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/1.jpg)
Microservices in a Streaming World
![Page 2: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/2.jpg)
There are many good reasons for building service-based systems
• Loose Coupling
• Bounded Contexts
• Autonomy
• Ease of scaling
• Composability
![Page 3: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/3.jpg)
But when we do, we’re building a distributed system
![Page 4: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/4.jpg)
This can be a bit tricky
![Page 5: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/5.jpg)
Monolithic & Centralised Approaches
Shared, mutable state
![Page 6: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/6.jpg)
Decentralisation
![Page 7: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/7.jpg)
![Page 8: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/8.jpg)
Stream Processing is a bit different
batch analytics => real time => at scale => accurately
![Page 9: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/9.jpg)
and comes with an interesting toolset
![Page 10: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/10.jpg)
Stream Processing Toolset
Business Applications
![Page 11: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/11.jpg)
Some fundamental patterns of distributed
systems
![Page 12: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/12.jpg)
Request / Response
![Page 13: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/13.jpg)
Mediator / Workflow
Request/Response
![Page 14: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/14.jpg)
Event Driven
Async / Fire and Forget
![Page 15: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/15.jpg)
Event Based
• Simple
• Synchronous
• Event Driven
• Good decoupling
• Requires Broker
• Fire & Forget
• Polling
• Full decoupling
Request/Response vs.
![Page 16: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/16.jpg)
SOA / Microservices
Message Broker
Event Based Request/Response
![Page 17: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/17.jpg)
Combinations
Event- Based
Request/Response
![Page 18: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/18.jpg)
CombinationsWithdraw
£100AccountService
GeneralLedger
CustomerStatements
FraudDetection
Check Funds
Async Message Broker
I need moneyReST
![Page 19: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/19.jpg)
Services generally eschew shared, mutable state
![Page 20: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/20.jpg)
How do we put these things together?
![Page 21: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/21.jpg)
Request/Response
![Page 22: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/22.jpg)
Request/Response
Request
Response
ReST
![Page 23: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/23.jpg)
Request/Response + Registry
Registry
Request
Response
ReST
![Page 24: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/24.jpg)
Asynchronous and Event-Based
Communication
![Page 25: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/25.jpg)
Queues
![Page 26: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/26.jpg)
Point to Point
Service A Service B
![Page 27: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/27.jpg)
Load Balancing
Instance 2
Instance 1
Single message allocation has scalability issues
![Page 28: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/28.jpg)
Batched AllocationInstance 1
Instance 2
Throughput!
![Page 29: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/29.jpg)
Lose Ordering Guarantees
Fail!
Instance 1
Instance 2
![Page 30: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/30.jpg)
Topics
![Page 31: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/31.jpg)
Topics are Broadcast
Consumer
Consumer
Broker broadcast
![Page 32: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/32.jpg)
Topics Retain Ordering
Trades
Buys
Sells
Broker Instance 1
Instance 2
![Page 33: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/33.jpg)
Even when services fail
Trades
Buys
Sells
Fail!Broker
We retain ordering, but we have to detect & reprovision
Instance 1
Instance 2
![Page 34: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/34.jpg)
A Few Implications
![Page 35: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/35.jpg)
Queues Lose Ordering Guarantees at Scale
Fail!
Worker 1
Worker 2
![Page 36: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/36.jpg)
Trades
Buys
Sells
Topics don’t provide availability
Broker
![Page 37: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/37.jpg)
Trades
Buys
Sells
Messages are Transient
Broker
![Page 38: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/38.jpg)
Is there another way?
![Page 39: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/39.jpg)
A Distributed LogKafka is one example
![Page 40: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/40.jpg)
Think back to the queue example
Batch
Batch
![Page 41: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/41.jpg)
Shard on the way in
![Page 42: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/42.jpg)
Each shard is a queue
Strong Ordering (in shard). Good concurrency.
![Page 43: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/43.jpg)
Each consuming service is assigned a “personal set” of queues
each little queue is sent to only one service in a group
![Page 44: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/44.jpg)
Services instances naturally rebalance on failure
Service instance dies, data is redirected, ordering guarantees remain
![Page 45: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/45.jpg)
Very Scalable, Very High Throughput
Sharded In, Sharded Out
![Page 46: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/46.jpg)
Reduces to a globally ordered queue
![Page 47: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/47.jpg)
Fault Tolerance
![Page 48: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/48.jpg)
The Log
Single seek & scan
Appendonly
messages don’t need to be transient!
![Page 49: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/49.jpg)
Cleaning the Log
Delete old segments
![Page 50: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/50.jpg)
Cleaning the Log
Delete old versions that share the same key
K1K1K1K2K2
K2K1
V1
V1
V2V3
V2V4V3
![Page 51: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/51.jpg)
• Scalable multiprocessing
• Strong partition-based ordering
• Efficient data retention
• Always on
![Page 52: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/52.jpg)
So how is this useful for microservices?
![Page 53: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/53.jpg)
Build ‘Always On’ Services
Rely on Fault Tolerant Broker
![Page 54: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/54.jpg)
Load Balance Services
Load Balance Services(with strong ordering)
![Page 55: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/55.jpg)
Fault Tolerant Services
Services automatically fail over
(retaining ordering)
![Page 56: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/56.jpg)
Services can return back to old messages in the log
Rewind & Replay
![Page 57: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/57.jpg)
Compacted Topics are Interesting
K1K1K1K2K2
K2K1
V1
V1
V2V3
V2V4V3
![Page 58: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/58.jpg)
Lets take a little example
![Page 59: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/59.jpg)
Getting Exchange Rates
ExchangeRate
Service
USD/GBP = 0.71EUR/GBP = 0.77USD/INR = 67.7USD/AUD = 1.38EUR/JPY = 114.41…
Ineed
exchange rates!
![Page 60: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/60.jpg)
Option1: Request Response
rate for USD/GBP?
0.71
ExchangeRate
Service
I need
exchange rates!
![Page 61: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/61.jpg)
Option 2: Publish Subscribe
ExchangeRate
Service
Accumulate current state
ETL
Ineed
exchange rates!
![Page 62: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/62.jpg)
Option 3: Accumulate in Compacted Stream
ExchangeRate
Service
Get all exchange rates
Publish to clients
USD/GBP = 0.71EUR/GBP = 0.77USD/INR = 67.7USD/AUD = 1.38EUR/JPY = 114.41…
Broker retainslatest versions
Publish all rate events
![Page 63: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/63.jpg)
Is it a stream or is it a table?
transitory stateful
![Page 64: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/64.jpg)
Datasets can live in the broker!
trades books
risk resultsex-rates
![Page 65: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/65.jpg)
Service BackboneScalable, Fault Tolerant, Concurrent, Strongly Ordered, Stateful
![Page 66: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/66.jpg)
… lets add in stream processing
![Page 67: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/67.jpg)
Max(price)From orderswhere ccy=‘GBP’over 1 day windowemitting every second
What is stream processing?
Continuous Queries.
![Page 68: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/68.jpg)
What is stream processing engine?
DataIndex
QueryEngine
QueryEnginevs
DatabaseFinite, well defined source
Stream ProcessorInfinite, poorly defined source
![Page 69: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/69.jpg)
Windowing
For unordered or unpredictable streams
Sliding
Fixed(tumbling)
![Page 70: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/70.jpg)
Features: similar to database query engine
JoinFilterAggr-egate View
Window
![Page 71: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/71.jpg)
KStreams & KTables
stream
Compactedstream
Join
Streaming Data
Stored Data
KStream
KTable
![Page 72: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/72.jpg)
A little example…
![Page 73: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/73.jpg)
Buying Lunch Abroad
PaymentsService
ExchangeRates
Service
Buy
Notification Service
Amount in ££
$$
$$
Text Message: ££
$$
![Page 74: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/74.jpg)
Request-Response Option
PaymentsService
ExchangeRates
Service
Buy
Amount in ££
Join etc
Text Message: ££
Iterative join over the network
![Page 75: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/75.jpg)
ETL Option
PaymentsService
ExchangeRates
Service
Buy
Amount in ££
ETL
ETLJoin etc
Text Message: ££
![Page 76: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/76.jpg)
Stream Processor Option
PaymentsService
ExchangeRates
Service
Buy
StreamProcessor
joinetc
Text Message: ££
![Page 77: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/77.jpg)
Buying Lunch Abroad
Payments
ExchangeRates
Looks like a table
(compactedstream)
Looks like an infinite stream
KStream
KTable
![Page 78: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/78.jpg)
Buying Lunch Abroad
Payments
ExchangeRates
• Filter(ccy<>’GBP’)• Join on ccy• Calculate GBP• Send text message
buffering
![Page 79: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/79.jpg)
Local DB (fast joins)
Topic
CompactedTopic
KStream
pre-populate
![Page 80: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/80.jpg)
KTables can also be written to - they’re backed by the broker
Manage intermediary
state
KStream
KTable
Topic
CompactedTopic
![Page 81: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/81.jpg)
Scales Out (MPP)
![Page 82: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/82.jpg)
These tools are pretty handy
for managing decentralised services
![Page 83: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/83.jpg)
Talk our own data model
DataStream
View
Query
![Page 84: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/84.jpg)
Handle Unpredictability
9am 5pm
Late trades
![Page 85: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/85.jpg)
Joining Services
Payments
ExchangeRates
Join
![Page 86: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/86.jpg)
Duality between Stream and Table
Join
KStream
KTable
![Page 87: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/87.jpg)
More Complex Use CasesTrades Valuations
Books Customers
General Ledger
![Page 88: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/88.jpg)
trades books
risk resultsex-rates
Practical mechanism for managing data intensive, loosely coupled services
• Stateful streams live inside the Log
• Data extracted quickly!
• Fast, local joins, over large datasets
• HA pre-caching
• Manage intermediary state
• Just a simple library (over Kafka)
![Page 89: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/89.jpg)
There is much more to stream processingit is grounded in the world of big-data analytics
![Page 90: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/90.jpg)
Simple Approaches
Just a library (over Kafka)
![Page 91: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/91.jpg)
Keeping Services Consistent
![Page 92: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/92.jpg)
Big Global Bag of State in the Sky
Problem: No BGBSS
![Page 93: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/93.jpg)
How to you provide the accuracy of this
![Page 94: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/94.jpg)
In this?
![Page 95: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/95.jpg)
Centralised vs Federated
Centralised consistency model
Distributed consistency model
![Page 96: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/96.jpg)
One problem is failure
![Page 97: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/97.jpg)
Duplicate messages are inevitable
have I seen this before?
![Page 98: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/98.jpg)
Make Services Idempotent
try 1
try 2
try 3
try 4
![Page 99: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/99.jpg)
Stream processors have to solve this problem
![Page 100: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/100.jpg)
Exactly Once
not available in Kafka… yet
![Page 101: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/101.jpg)
So what do we have?
![Page 102: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/102.jpg)
Use Both Approaches
Event- Based
Request/Response
![Page 103: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/103.jpg)
Queued Delivery System
Ordered queue
![Page 104: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/104.jpg)
Scales Horizontally
![Page 105: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/105.jpg)
Scales Horizontally
![Page 106: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/106.jpg)
Scales Horizontally
![Page 107: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/107.jpg)
Scales Horizontally
![Page 108: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/108.jpg)
Built In Fault Tolerance
![Page 109: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/109.jpg)
Runs Always On
![Page 110: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/110.jpg)
For Services Too
![Page 111: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/111.jpg)
Scales Horizontally
Load Balance
![Page 112: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/112.jpg)
continue through failure
![Page 113: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/113.jpg)
Scales Horizontally
with history stored in the Log
![Page 114: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/114.jpg)
Scales Horizontally
Extending to any number of services
![Page 115: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/115.jpg)
Scales Horizontally
With any data throughput
![Page 116: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/116.jpg)
Scales Horizontally
With any data throughput
![Page 117: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/117.jpg)
Scales Horizontally
With any data throughput
![Page 118: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/118.jpg)
Scales Horizontally
powerful tools for slicing and dicing streams
![Page 119: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/119.jpg)
Scales Horizontally
the declarative processing of
datajoin
filteraggregate
![Page 120: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/120.jpg)
at any throughput
![Page 121: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/121.jpg)
Scales Horizontally
leveraging fast local
persistence
![Page 122: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/122.jpg)
Scales Horizontally
backed up to the log
![Page 123: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/123.jpg)
Scales Horizontally
easily join streaming services
![Page 124: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/124.jpg)
Blend KStreams
and KTables
![Page 125: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/125.jpg)
trades books
risk resultsex-rates
with data living in the stream
![Page 126: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/126.jpg)
but retaining loose coupling
trades books
risk resultsex-rates
![Page 127: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/127.jpg)
Scales Horizontally
with strong ordering and repeatability guarantees
(eventually)
![Page 128: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/128.jpg)
so…
![Page 129: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/129.jpg)
Microservices push us away from shared, mutable state
![Page 130: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/130.jpg)
Big Global Bag of State in the Sky
Away from BGBSS’s
![Page 131: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/131.jpg)
This means data is increasingly remote
![Page 132: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/132.jpg)
Sure, you can collect it all
copy copy
copy
copycopy
copy
copy
ETL
ETL
ETL ETL
ETL
ETL
![Page 133: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/133.jpg)
can be a lot of work
![Page 134: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/134.jpg)
Or you can look it all up
getget
get
getget
get
get
get, get, get, get
![Page 135: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/135.jpg)
but that doesn’t scale well(with system complexity or with data throughput)
![Page 136: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/136.jpg)
Better to embrace decentralistion
![Page 137: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/137.jpg)
We need a decentralised toolset to do this
trades books
risk resultsex-
rates
![Page 138: Microservices for a Streaming World](https://reader031.fdocuments.us/reader031/viewer/2022030317/587142a01a28ab55588b4c9f/html5/thumbnails/138.jpg)
Keep it simple, Keep it moving