FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date:...
Transcript of FoundationDB Technical Overview v2 - QCon · FoundationDB Technical Overview v2 Created Date:...
Evan Tschannen
�1
• Worked on FoundationDB for 8 years
• Touched every core component of the database
• Recently added multi-region asynchronous replication
Evan Tschannen
• https://www.foundationdb.org
• https://github.com/apple/foundationdb
• Open Source, Apache License (v2)
Why are there so many databases?
!4
!5
• Tension between performance, scalability, and consistency
• A wide variety of data models
Why are there so many databases?
FoundationDB Performance
• Organizes many instances of a single process database into a single database
• Read and write throughput 90% of the aggregate individual read and write throughput
• Single hop read latencies
• Four hop write latencies
FoundationDB is a…
• Highly performant,
• Scalable,
• Ordered key value store,
• That supports ACID transactions.
!8
A
B
C
Writer
Servers
Reader
Wait for 2 of 3 responses Wait for 2 of 3 responses
!9
A
B
C
Writer
Servers
Reader
Wait for 2 of 3 responses Wait for 2 of 3 responses
!10
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
!11
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
!12
A
B
C
Writer
Servers
Reader
Wait for 3 of 3 responses Wait for 1 of 3 responses
Versions(Stored in a different database)
D
0: A,B,C 101: B,C,D
!13
Coordinators
Storage Servers
ZooKeeper (Failure Handling)
SQLite (Reads)
Transaction Logs
FoundationDB (Writes)
!14
Coordinators
Storage Servers Transaction Logs
Cluster Controller
!15
Coordinators
Storage Servers Transaction Logs
Cluster Controller
!16
Get Read Version
ReadWrite
Commit
FoundationDB APIVersions
200
400
100 A : apple(200)
(A, 200)
(B, banana)
!17
Get Read Version
ReadWrite
Commit
FoundationDB APIVersions
200
400
100 A : apple(200)
(A, 200)
(B, banana)300 A : orange
!18
Coordinators
Storage Servers
Cluster Controller
Master
Proxies Resolvers Transaction Logs
!19
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
!20
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(B : banana)
Data Distribution
!21
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
!22
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
!23
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(B : banana)
!24
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(200)
(A, 200)
(400)
(B : banana)
!25
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
!26
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(400)
(300)
(200)
!27
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
(400)
(400)
(300)
(200)
!28
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
!29
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
!30
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
!31
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
Data Distribution
!32
Coordinators
Cluster Controller
Master
Proxies Resolvers Transaction LogsStorage Servers
Get Read Version
Reads
Writes
Commit
FoundationDB Performance
• Organizes many instances of a single process database into a single database
• Read and write throughput 90% of the aggregate individual read and write throughput
• Single hop read latencies
• Four hop write latencies
Does it work?
• Run the entire distributed database in a single process
• Inject random failures
• Check correctness
• Reproduce errors
https://www.foundationdb.org
!35