FleetDBA Schema-Free Database in Clojure

Mark McGranaghan

January 23, 2010

Motivation

Walkthrough

Implementation

Motivation

optimize for agile development

Motivation

optimize for agile development

Walkthrough

I Client Quickstart

I Basic Queries

I Concurrency Tools

Client Quickstart

(use ’fleetdb.client)

(def client (connect))

(client ["ping"])

=> "pong"

Insert

(client

["insert" "accounts"

{"id" 1 "owner" "Eve" "credits" 100}])

=> 1

Insert

(client

["insert" "accounts"

{"id" 1 "owner" "Eve" "credits" 100}])

=> 1

Select

(client

["select" "accounts" {"where" ["=" "id" 1]}])

=> [{"id" 1 "owner" "Eve" "credits" 100}]

Select

(client

["select" "accounts" {"where" ["=" "id" 1]}])

=> [{"id" 1 "owner" "Eve" "credits" 100}]

Select with Conditions

(client

["select" "accounts"

{"where" [">=" "credits" 150]}])

Select with Order and Limit

(client

["select" "accounts"

{"order" ["credits" "asc"] "limit" 2}])

Update

(client

["update" "accounts" {"credits" 105}

{"where" ["=" "owner" "Eve"]}])

Explain (I)

(client

["explain" ["select" "accounts"

{"where" ["=" "owner" "Eve"]}]])

=> ["filter" ["=" "owner" "Eve"]

["collection-scan" "accounts"]]

Explain (I)

(client

["explain" ["select" "accounts"

{"where" ["=" "owner" "Eve"]}]])

=> ["filter" ["=" "owner" "Eve"]

["collection-scan" "accounts"]]

Create Index

(client

["create-index" "accounts" "owner"])

Explain (II)

(client

["explain" ["select" "accounts"

{"where" ["=" "owner" "Eve"]}]])

=> ["index-lookup" ["accounts" "owner" "Eve"]]]

Multi-Write

(client

["multi-write"

[write-query-1 write-query-2 ...]])

=> [result-1 result-2 ...]

Multi-Write

(client

["multi-write"

[write-query-1 write-query-2 ...]])

=> [result-1 result-2 ...]

Multi-Write

(client

["multi-write"

[write-query-1 write-query-2 ...]])

=> [result-1 result-2 ...]

Checked-Write

(client

["checked-write"

read-query

expected-read-result

write-query])

=> [true write-result]

=> [false actual-read-result]

Checked-Write

(client

["checked-write"

read-query

expected-read-result

write-query])

=> [true write-result]

=> [false actual-read-result]

Checked-Write

(client

["checked-write"

read-query

expected-read-result

write-query])

=> [true write-result]

=> [false actual-read-result]

Checked-Write

(client

["checked-write"

read-query

expected-read-result

write-query])

=> [true write-result]

=> [false actual-read-result]

Implementation

I Background

I Organization

I Key ideas

Background

I http://clojure.org/data_structures

I http://clojure.org/sequences

I http://clojure.org/state

Organization

I fleetdb.core: pure functions

I fleetdb.embedded: identity and durability

I fleetdb.server: network interface

Organization

I fleetdb.core: pure functions

I fleetdb.embedded: identity and durability

I fleetdb.server: network interface

Organization

I fleetdb.core: pure functions

I fleetdb.embedded: identity and durability

I fleetdb.server: network interface

Organization

I fleetdb.core: pure functions

I fleetdb.embedded: identity and durability

I fleetdb.server: network interface

fleetdb.core

I Pure functions

I Databases as Clojure data structures

I Read: db + query → result

I ‘Write’: db + query → result + new db

fleetdb.core

I Pure functions

I Databases as Clojure data structures

I Read: db + query → result

I ‘Write’: db + query → result + new db

fleetdb.core

I Pure functions

I Databases as Clojure data structures

I Read: db + query → result

I ‘Write’: db + query → result + new db

fleetdb.core

I Pure functions

I Databases as Clojure data structures

I Read: db + query → result

I ‘Write’: db + query → result + new db

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Planner

I Implements declarative queries

I Database + query → plan

["select" "accounts"

{"where" ["=" "owner" "Eve"]}]

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

["index-lookup" ["accounts" "owner" "Eve"]]

fleetdb.core Query Executor

I Database + plan → result

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

(filter (fn [r] (= (r "owner") "Eve"))

(vals (:rmap (db "accounts"))))

fleetdb.core Query Executor

I Database + plan → result

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

(filter (fn [r] (= (r "owner") "Eve"))

(vals (:rmap (db "accounts"))))

fleetdb.core Query Executor

I Database + plan → result

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

(filter (fn [r] (= (r "owner") "Eve"))

(vals (:rmap (db "accounts"))))

fleetdb.core Query Executor

I Database + plan → result

["filter" ["=" "owner" "Eve"]

["record-scan" "accounts"]]

(filter (fn [r] (= (r "owner") "Eve"))

(vals (:rmap (db "accounts"))))

fleetdb.embedded

I Wraps fleetdb.core

I Adds identity and durability

I Databases in atoms

I Append-only log

fleetdb.embedded

I Wraps fleetdb.core

I Adds identity and durability

I Databases in atoms

I Append-only log

fleetdb.embedded

I Wraps fleetdb.core

I Adds identity and durability

I Databases in atoms

I Append-only log

fleetdb.embedded

I Wraps fleetdb.core

I Adds identity and durability

I Databases in atoms

I Append-only log

fleetdb.embedded Read Path

I Dereference database atom

I Pass to fleetdb.core

I Return result

fleetdb.embedded Read Path

I Dereference database atom

I Pass to fleetdb.core

I Return result

fleetdb.embedded Read Path

I Dereference database atom

I Pass to fleetdb.core

I Return result

fleetdb.embedded Read Path

I Dereference database atom

I Pass to fleetdb.core

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.embedded Write Path

I Enter fair lock

I Dereference database atom

I Pass to fleetdb.core

I Append query to log

I Swap in new database value

I Return result

fleetdb.server

I Wraps fleetdb.embedded

I Adds JSON client API

fleetdb.server

I Wraps fleetdb.embedded

I Adds JSON client API

Aside: Source Size

Lines of Codecore 630embedded 130server 120lint 280utilities 140total 1300

Aside: Source Size

Lines of Codecore 630embedded 130server 120lint 280utilities 140total 1300

Takeaways

I Clojure’s data structures are awesome

I Clojure is viable for infrastructure software

I Try FleetDB!

Takeaways

I Clojure’s data structures are awesome

I Clojure is viable for infrastructure software

I Try FleetDB!

Takeaways

I Clojure’s data structures are awesome

I Clojure is viable for infrastructure software

I Try FleetDB!

Takeaways

I Clojure’s data structures are awesome

I Clojure is viable for infrastructure software

I Try FleetDB!

Learn More

I http://FleetDB.org

I http://github.com/mmcgrana