FleetDB: A Schema-Free Database in Clojure
-
Upload
mark-mcgranaghan -
Category
Technology
-
view
2.182 -
download
0
Transcript of FleetDB: A Schema-Free Database in Clojure
FleetDBA Schema-Free Database in Clojure
Mark McGranaghan
January 23, 2010
Motivation
Walkthrough
Implementation
Motivation
optimize for agile development
Motivation
optimize for agile development
Walkthrough
I Client Quickstart
I Basic Queries
I Concurrency Tools
Client Quickstart
(use ’fleetdb.client)
(def client (connect))
(client ["ping"])
=> "pong"
Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])
=> 1
Insert
(client
["insert" "accounts"
{"id" 1 "owner" "Eve" "credits" 100}])
=> 1
Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])
=> [{"id" 1 "owner" "Eve" "credits" 100}]
Select
(client
["select" "accounts" {"where" ["=" "id" 1]}])
=> [{"id" 1 "owner" "Eve" "credits" 100}]
Select with Conditions
(client
["select" "accounts"
{"where" [">=" "credits" 150]}])
Select with Order and Limit
(client
["select" "accounts"
{"order" ["credits" "asc"] "limit" 2}])
Update
(client
["update" "accounts" {"credits" 105}
{"where" ["=" "owner" "Eve"]}])
Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["filter" ["=" "owner" "Eve"]
["collection-scan" "accounts"]]
Explain (I)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["filter" ["=" "owner" "Eve"]
["collection-scan" "accounts"]]
Create Index
(client
["create-index" "accounts" "owner"])
Explain (II)
(client
["explain" ["select" "accounts"
{"where" ["=" "owner" "Eve"]}]])
=> ["index-lookup" ["accounts" "owner" "Eve"]]]
Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]
Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]
Multi-Write
(client
["multi-write"
[write-query-1 write-query-2 ...]])
=> [result-1 result-2 ...]
Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]
Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]
Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]
Checked-Write
(client
["checked-write"
read-query
expected-read-result
write-query])
=> [true write-result]
=> [false actual-read-result]
Implementation
I Background
I Organization
I Key ideas
Background
I http://clojure.org/data_structures
I http://clojure.org/sequences
I http://clojure.org/state
Organization
I fleetdb.core: pure functions
I fleetdb.embedded: identity and durability
I fleetdb.server: network interface
Organization
I fleetdb.core: pure functions
I fleetdb.embedded: identity and durability
I fleetdb.server: network interface
Organization
I fleetdb.core: pure functions
I fleetdb.embedded: identity and durability
I fleetdb.server: network interface
Organization
I fleetdb.core: pure functions
I fleetdb.embedded: identity and durability
I fleetdb.server: network interface
fleetdb.core
I Pure functions
I Databases as Clojure data structures
I Read: db + query → result
I ‘Write’: db + query → result + new db
fleetdb.core
I Pure functions
I Databases as Clojure data structures
I Read: db + query → result
I ‘Write’: db + query → result + new db
fleetdb.core
I Pure functions
I Databases as Clojure data structures
I Read: db + query → result
I ‘Write’: db + query → result + new db
fleetdb.core
I Pure functions
I Databases as Clojure data structures
I Read: db + query → result
I ‘Write’: db + query → result + new db
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Planner
I Implements declarative queries
I Database + query → plan
["select" "accounts"
{"where" ["=" "owner" "Eve"]}]
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
["index-lookup" ["accounts" "owner" "Eve"]]
fleetdb.core Query Executor
I Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))
fleetdb.core Query Executor
I Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))
fleetdb.core Query Executor
I Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))
fleetdb.core Query Executor
I Database + plan → result
["filter" ["=" "owner" "Eve"]
["record-scan" "accounts"]]
(filter (fn [r] (= (r "owner") "Eve"))
(vals (:rmap (db "accounts"))))
fleetdb.embedded
I Wraps fleetdb.core
I Adds identity and durability
I Databases in atoms
I Append-only log
fleetdb.embedded
I Wraps fleetdb.core
I Adds identity and durability
I Databases in atoms
I Append-only log
fleetdb.embedded
I Wraps fleetdb.core
I Adds identity and durability
I Databases in atoms
I Append-only log
fleetdb.embedded
I Wraps fleetdb.core
I Adds identity and durability
I Databases in atoms
I Append-only log
fleetdb.embedded Read Path
I Dereference database atom
I Pass to fleetdb.core
I Return result
fleetdb.embedded Read Path
I Dereference database atom
I Pass to fleetdb.core
I Return result
fleetdb.embedded Read Path
I Dereference database atom
I Pass to fleetdb.core
I Return result
fleetdb.embedded Read Path
I Dereference database atom
I Pass to fleetdb.core
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.embedded Write Path
I Enter fair lock
I Dereference database atom
I Pass to fleetdb.core
I Append query to log
I Swap in new database value
I Return result
fleetdb.server
I Wraps fleetdb.embedded
I Adds JSON client API
fleetdb.server
I Wraps fleetdb.embedded
I Adds JSON client API
Aside: Source Size
Lines of Codecore 630embedded 130server 120lint 280utilities 140total 1300
Aside: Source Size
Lines of Codecore 630embedded 130server 120lint 280utilities 140total 1300
Takeaways
I Clojure’s data structures are awesome
I Clojure is viable for infrastructure software
I Try FleetDB!
Takeaways
I Clojure’s data structures are awesome
I Clojure is viable for infrastructure software
I Try FleetDB!
Takeaways
I Clojure’s data structures are awesome
I Clojure is viable for infrastructure software
I Try FleetDB!
Takeaways
I Clojure’s data structures are awesome
I Clojure is viable for infrastructure software
I Try FleetDB!
Learn More
I http://FleetDB.org
I http://github.com/mmcgrana