How to write your database: the story about Event Store
-
Upload
victor-haydin -
Category
Technology
-
view
5.361 -
download
1
description
Transcript of How to write your database: the story about Event Store
![Page 1: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/1.jpg)
HOW TO WRITE YOUR
DATABASEThe story about
By Victor Haydin, Head of R&D, ELEKS
![Page 2: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/2.jpg)
Agenda
1.Problem definition2.Event Store: introduction3.Under the hood4.Lessons learned5.Q&A
![Page 3: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/3.jpg)
Few notes before we start1.This story is about the development, technology
and lessons learned, not a product2.Neither about Event Sourcing3.Outsourcing point of view4.I was there on pre-sales and initiation phases.
When the team was formed I left the project.
![Page 4: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/4.jpg)
Survey
![Page 5: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/5.jpg)
How many of you can understand code?
![Page 6: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/6.jpg)
Event Sourcing?
![Page 7: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/7.jpg)
Event Store?
![Page 8: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/8.jpg)
SEDA?
![Page 9: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/9.jpg)
140 characters long definition:
Event sourcing - persisting entities by appending all business events to transaction log. To rebuild the state, we replay this log.
![Page 10: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/10.jpg)
ExampleBankAccount
GotSallary: +100GotCashFromAtm: -10MadeInternetPayment: -15MadeTerminalPayment: -5PutCashToAccount: +20
Account balance: 100-10-15-5+20=90
![Page 11: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/11.jpg)
Event SourcingPros Cons
Performance Performance
Simplification Versioning
Audit Trail Querying
Integration with other systems Not common approach
Troubleshooting Better tastes with CQRS
Fixing errors
Testing
Flexibility
![Page 12: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/12.jpg)
![Page 13: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/13.jpg)
The marketing says
![Page 14: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/14.jpg)
and time-series data
![Page 15: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/15.jpg)
for single-node configuration
![Page 16: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/16.jpg)
AtomPub protocol for HTTP
![Page 17: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/17.jpg)
33000 writes per second on single node
![Page 18: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/18.jpg)
Master-Slave replication for Premium version
![Page 19: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/19.jpg)
I’ll talk more about Mono later
![Page 20: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/20.jpg)
web-based
![Page 21: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/21.jpg)
Client API1.Append to Stream2.Read Events3.Subscribe4.Transactions5.Stream Metadata6.Delete Stream7.System Settings8.Connection Lifecycle
![Page 22: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/22.jpg)
Under the hood
![Page 23: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/23.jpg)
Disclaimer
![Page 24: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/24.jpg)
SEDA
![Page 25: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/25.jpg)
Time to take a look at code
![Page 26: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/26.jpg)
SEDA outcome1.Very few locks in code2.Simple state management3.Easy to understand system composition4.Easy to extend5.Easy to monitor (DEMO)
![Page 27: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/27.jpg)
License is MIT, code is available on github. Enjoy!
![Page 28: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/28.jpg)
Storage Engine
![Page 29: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/29.jpg)
All events are stored in one transaction log (splitted into chunks)
![Page 30: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/30.jpg)
Writer Chaser Readers Scavenger
Storage pipeline
![Page 31: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/31.jpg)
Writer1.Single-threaded2.Append-only3.Write data in chunks4.Flush depending on load and settings
![Page 32: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/32.jpg)
Chaser1.Checks what writer has appended to log and
sends acknowledgements 2.Updates index
![Page 33: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/33.jpg)
Readers1.Read events from transaction log with help of
Read Index
![Page 34: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/34.jpg)
Read Index1.Maps stream name and version to position in
transaction log2.Index = MemTable + PTables3.MemTable = Dictionary of Lists4.Index entry – record of fixed length5.PTable = file with sorted sequence of entries
and cached midpoints6.Binary search FTW!
![Page 35: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/35.jpg)
Scavenger1.Background process that go through transaction
log chunks and clean it from obsolete data (e.g. deleted streams, $maxCount, $maxAge etc.)
2.Simply writes data to a new chunk file3.Last one out shut the lights off
![Page 36: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/36.jpg)
High Availability
![Page 37: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/37.jpg)
High Availability1.Master-Slave replication2.Automatic master elections (N/2 + 1 quorum)3.Wins the one with most actual state4.Write always to master, read from master or
slave, depending on client’s choice 5.Byte stream replication6.Available in commercial version
![Page 38: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/38.jpg)
Projections
![Page 39: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/39.jpg)
Projection – the process of taking an event stream and converting it to some other form (e.g. another event stream or state object)
![Page 40: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/40.jpg)
Projections Engine1.Built-in query language – Javascript2.Based on Google’s v8
![Page 41: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/41.jpg)
Demo: Projections Engine
![Page 42: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/42.jpg)
Lessons Learned
![Page 43: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/43.jpg)
Immutability is good
![Page 44: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/44.jpg)
Pure async is fast,although hard for development
![Page 45: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/45.jpg)
Lockless programming is hard
![Page 46: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/46.jpg)
Distributed programming is even harder
![Page 47: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/47.jpg)
Debugger is almost useless
![Page 48: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/48.jpg)
Unit-testing is a limited tool
![Page 49: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/49.jpg)
Verbose logs rock
![Page 50: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/50.jpg)
Real-life testingrocks like nothing else
![Page 51: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/51.jpg)
![Page 52: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/52.jpg)
Bugs in .NET[System.Security.SecuritySafeCritical]public virtual void Flush(Boolean flushToDisk) {
// This code is duplicated in Disposeif (_handle.IsClosed) __Error.FileNotOpen();if (_writePos > 0) {FlushWrite(false);if (flushToDisk) {
if (!Win32Native.FlushFileBuffers(_handle)) {__Error.WinIOError();
}}
}else if (_readPos < _readLen && CanSeek) {FlushRead();
}_readPos = 0;_readLen = 0;
}
*Appears to be fixed in 4.5
![Page 53: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/53.jpg)
Bugs in Mono
TCP/IP Stack deadlocks Concurrent Stack/Queue
XML Serializer
File Stream issuesGeneral slowness:
20K w/s
![Page 54: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/54.jpg)
Disk caches: fake flush and reordering
*By default, can be disabled
![Page 55: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/55.jpg)
Mind the HDD/SSD difference
*It is possible to optimize for both
![Page 56: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/56.jpg)
There is a long way to simplicity
![Page 57: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/57.jpg)
Experiments:The way it works
![Page 58: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/58.jpg)
Further plans1.Horizontal scalability2.Hosted service3.Adding other features
![Page 59: How to write your database: the story about Event Store](https://reader034.fdocuments.us/reader034/viewer/2022051211/5562df1cd8b42a63498b58e3/html5/thumbnails/59.jpg)
Links1. GetEventStore.com – official web site
2. GitHub.com/EventStore/EventStore – source code
3. OreDev.org/2012/sessions/a-deep-look-into-the-event-store – A deep look into The Event Store by Greg Young
4. MartinFowler.com/eaaDev/EventSourcing.html – Event Sourcing overview by Martin Fowler
5. Google <Event Sourcing|Event Store>