The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford...
Transcript of The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford...
![Page 1: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/1.jpg)
The Power of the Log LSM & Append Only Data Structures
Ben Stopford Confluent Inc
![Page 2: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/2.jpg)
@benstopford
![Page 3: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/3.jpg)
The Log Connectors Connectors
Producer Consumer
Streaming Engine
Kafka: a Streaming Platform
![Page 4: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/4.jpg)
KAFKA’s Distributed Log
Linear Scans Append Only
![Page 5: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/5.jpg)
Messaging is a Log-Shaped Problem
Linear Scans Append Only
![Page 6: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/6.jpg)
Not all problems are Log-Shaped
![Page 7: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/7.jpg)
Many problems benefit from being addressed in a “log-shaped” way
![Page 8: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/8.jpg)
Supporting Lookups
![Page 9: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/9.jpg)
Lookups in a log
Head Tail
![Page 10: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/10.jpg)
Trees provide Selectivity
bob dave fred hary mike steve vince
Index
![Page 11: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/11.jpg)
But the overarching structure implies Dispersed Writes
bob dave fred hary mike steve vince
Random IO
![Page 12: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/12.jpg)
Log Structured Merge Trees 1996
![Page 13: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/13.jpg)
Used in a range of modern databases
• BigTable
• HBase
• LevelDB
• SQLite4
• RocksDB
• MongoDB
• WiredTiger
• Cassandra
• MySQL
• InfluxDB ...
![Page 14: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/14.jpg)
If a systems have a natural grain, it is one formed of sequential
operations which favour locality
![Page 15: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/15.jpg)
Caching & Prefetching
L3 cache L2 cache
L1 cache
Pre-fetch is your friend
CPU Caches
Page Cache
Application-level caching
Disk Controller
![Page 16: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/16.jpg)
Write efficiency comes from amortising writes into sequential
operations
![Page 17: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/17.jpg)
Taken from ACMQueue: The Pathologies of Big Data
![Page 18: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/18.jpg)
So if we go against the grain of the system, RAM can actually be
slower than disk
![Page 19: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/19.jpg)
Going against the grain means dispersed operations that break locality
Poor Locality Good Locality
![Page 20: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/20.jpg)
The beauty of the log lies in its sequentially
Linear Scans Append Only
![Page 21: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/21.jpg)
LSM is about re-imagining search as as a “log-shaped” problem
![Page 22: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/22.jpg)
Arrange writes to be Append Only
Append Only Journal
(Sequential IO)
Update in Place Ordered File (Random IO)
Bob = Carpenter
Bob = Carpenter
Bob = Cabinet Maker
Bob = Cabinet Maker
![Page 23: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/23.jpg)
Avoid dispersed writes
![Page 24: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/24.jpg)
Simple LSM
![Page 25: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/25.jpg)
Writes are collected in memory Writes
sort
write to disk
older files
small index file
RAM
![Page 26: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/26.jpg)
When enough have buffered, sort. Writes
write to disk
older files
small index file
Batched sorted
RAM
![Page 27: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/27.jpg)
Write the sorted file to disk Writes
write to disk
older files
Small, sorted immutable file
Batched sorted
![Page 28: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/28.jpg)
Repeat... Writes
write to disk
Older files New files
Batched sorted
![Page 29: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/29.jpg)
Batching -> Fast Sequential IO Writes
write to disk
Older files New files
Batched
Sorted memtable
![Page 30: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/30.jpg)
That’s the core write path
![Page 31: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/31.jpg)
What about reads?
![Page 32: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/32.jpg)
Search reverse-chronologically
older files
newer files
(1) Is “bob” here?
(2) Is “bob” here?
(3) Is “bob” here?
(4) Is “bob” here?
![Page 33: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/33.jpg)
Worst Case We consult every file
![Page 34: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/34.jpg)
We might have a lot of files!
![Page 35: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/35.jpg)
LSM naturally optimises for writes, over reads
This is a reasonable tradeoff to make
![Page 36: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/36.jpg)
Optimizing reads is easier than optimising writes
![Page 37: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/37.jpg)
Optimisation 1 Bound the number of files
![Page 38: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/38.jpg)
Create levels
Level-0
Level-1
![Page 39: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/39.jpg)
Separate thread merges old files, de-duplicating them.
Level-0
Level-1
![Page 40: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/40.jpg)
Separate thread merges old files, de-duplicating them.
Level-0
Level-1
![Page 41: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/41.jpg)
Merging process is reminiscent of merge sort
![Page 42: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/42.jpg)
Take this further with levels
Level-0
Level-1
Level-2
Level-3 Memtable
![Page 43: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/43.jpg)
But single reads still require many individual lookups:
• Number of searches: – 1 per base level – 1 per level above
![Page 44: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/44.jpg)
Optimisation 2 Caching & Friends
![Page 45: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/45.jpg)
Add Memory i.e. More Caching / Pre-fetch
![Page 46: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/46.jpg)
Read Ahead & Prefetch
L3 cache L2 cache
L1 cache
Pre-fetch is your friend
Page Cache
Disk Controller
![Page 47: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/47.jpg)
If only there was a more efficient way to avoid searching each file!
![Page 48: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/48.jpg)
Elven Magic?
![Page 49: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/49.jpg)
Bloom Filters Answers the question:
Do I need to look in this file to find the value for this key?
Size -> probability of false positive Key
Hash Function
Bit Set
![Page 50: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/50.jpg)
Bloom Filters • Space efficient, probabilistic
data structure
• As keyspace grows: – p(collision) increases
– Index size is fixed
![Page 51: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/51.jpg)
Many more degrees of freedom for optimising reads
RAM
Disk
file metadata & bloom filter
![Page 52: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/52.jpg)
Log Structured Merge Trees • A collection of small, immutable indexes
• All sequential operations, de-duplicate by merging files
• Index/Bloom in RAM to increase read performance
![Page 53: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/53.jpg)
Subtleties • Writes are 1 x IO (blind writes) , rather than 2 x IO’s
(read + modify)
• Batching writes decreases write amplification. In trees leaf pages must be updated.
![Page 54: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/54.jpg)
Immutability => Simpler locking semantics
Only memtable is mutable
![Page 55: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/55.jpg)
Does it work? Lots of real world examples
![Page 56: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/56.jpg)
Measureable in the real world
• Innodb vs MyRocks results, taken from Mark Callaghan’s blog: http://bit.ly/2mhWT7p
• There are many subtleties. Take all benchmarks with a pinch of salt.
![Page 57: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/57.jpg)
Elements of Beauty • Reframing the problem to be Log-Centric. To go with
the grain of the system.
• Optimise for the harder problem
• Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
![Page 58: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/58.jpg)
Applies in many other areas • Sequentiality – Databases: write ahead logs
– Columnar databases: Merge Joins
– Kafka • Immutability – Snapshot isolation over explicit locking.
– Replication (state machines replication)
![Page 59: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/59.jpg)
Log-Centric Approaches Work in Applications too
![Page 60: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/60.jpg)
Event Sourcing • Journaling of
state changes
• No “update in place”
Object
Journal
+ 10.36 - 12.12 + 23.70 + 13.33
![Page 61: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/61.jpg)
CQRS Client
Command Query
Write Optimised
Read Optimised
log
![Page 62: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/62.jpg)
How Applications or Services share state
![Page 63: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/63.jpg)
Log-Centric Services
Writer Read-Replica
Read-Replica
Read-Replica
Writes are localised to a single service
![Page 64: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/64.jpg)
Log-Centric Services
Writer Read-Replica
Read-Replica
Read-Replica Immutable log
![Page 65: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/65.jpg)
Log-Centric Services
Writer Read-Replica
Read-Replica
Read-Replica Many, independent read replicas
![Page 66: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/66.jpg)
Elements of Beauty • Reframing the problem to be Log-Centric. To go with
the grain of the system.
• Optimise for the harder problem
• Compartmentalises writes (coordination) to a single point. Reads -> immutable structures.
![Page 67: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/67.jpg)
Decentralised Design In both database design as well as in
application development
![Page 68: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/68.jpg)
The Log is the central building block
Pushes us towards the natural grain of the system
![Page 69: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/69.jpg)
The Log A single unifying abstraction
![Page 70: The Power of the Log · The Power of the Log LSM & Append Only Data Structures Ben Stopford Confluent Inc . @benstopford . Connectors The Log Connectors Producer Consumer Streaming](https://reader036.fdocuments.us/reader036/viewer/2022071109/5fe4b134a4c9c96eaa5dbd78/html5/thumbnails/70.jpg)
References LSM:
• benstopford.com/2015/02/14/log-structured-merge-trees/
• smalldatum.blogspot.co.uk/2017/02/using-modern-sysbench-to-compare.html
• www.quora.com/How-does-the-Log-Structured-Merge-Tree-work
• bLSM paper: http://bit.ly/2mT7Vje
Other
• Pat Helland (Immutability) cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
• Peter Ballis (Coordination Avoidance): http://bit.ly/2m7XxnI
• Jay Kreps: I Heart Logs (O’Reilly 2014)
• The Data Dichotomy: http://bit.ly/2hk9c2K