The algebra of library design

The algebra of library design

#cljsyd - October 2014

Leonardo Borges @leonardo_borges

www.leonardoborges.com www.thoughtworks.com

Algebra?

More specifically…

• Algebraic structures, studied in Abstract Algebra • a set with one or more operations on it

• Category theory is another way to study these structures

Overview

• The problem with Clojure futures • A better futures library • Functors, Applicative Functors and Monads • What about core.async (and others?)

The beginning…

(def age (future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")!! (def double-age (* 2 @age))! (prn "age, doubled: " double-age)!! (prn "doing something else important...")

Do you see any issues?

The beginning…

(def age (future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")!! (def double-age (* 2 @age))! (prn "age, doubled: " double-age)!! (prn "doing something else important...")!! ;; "fetching age..."! ;; "age, doubled: " 62! ;; "doing something else important..."!

It looks like we would like to execute something once the Future has completed

A first glance at imminent (require '[imminent.core :as i])!! (def age (i/future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! (def double-age (i/map age #(* % 2)))! (i/on-success double-age #(prn "age, doubled: " %))!! (prn "doing something else important...")

A first glance at imminent (require '[imminent.core :as i])!! (def age (i/future (do (Thread/sleep 2000)! 31)))! (prn "fetching age...")! (def double-age (i/map age #(* % 2)))! (i/on-success double-age #(prn "age, doubled: " %))!! (prn "doing something else important...")!! ;; "fetching age..."! ;; "doing something else important..."! ;; "age, doubled: " 62!

Functor

class Functor f where! fmap :: (a -> b) -> f a -> f b

Another example

(def age (future (do (Thread/sleep 2000)! 31)))! (def name (future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (prn (format "%s is %s years old" @name @age))! (prn "more important things going on...")

Same as before, only this time we want to execute the code once both futures have completed

Rewriting it in imminent (def age (i/future (do (Thread/sleep 2000)! 31)))! (def name (i/future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (def both (i/sequence [name age]))! (i/on-success both ! (fn [[name age]]! (prn (format "%s is %s years old" name age))))!! (prn "more important things going on...")!!

Rewriting it in imminent (def age (i/future (do (Thread/sleep 2000)! 31)))! (def name (i/future (do (Thread/sleep 2000)! "Leonardo")))! (prn "fetching name...")! (def both (i/sequence [name age]))! (i/on-success both ! (fn [[name age]]! (prn (format "%s is %s years old" name age))))!! (prn "more important things going on...")!! ;; "fetching name..."! ;; "more important things going on..."! ;; "Leonardo is 31 years old"!

Monad

class Monad m where! (>>=) :: m a -> (a -> m b) -> m b! return :: a -> m a

Monad

class Monad m where! (>>=) :: m a -> (a -> m b) -> m b! return :: a -> m a

(>>=) is also called bind, flatmap, selectMany and mapcat…

Monad - derived functions

liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r


liftM2 :: (Monad m) => (a1 -> a2 -> r) -> m a1 -> m a2 -> m r

(defn mlift2! [f]! (fn [ma mb]! (flatmap ma! (fn [a]! (flatmap mb! (fn [b]! (pure (f a b))))))))


sequence :: Monad m => [m a] -> m [a]


sequence :: Monad m => [m a] -> m [a]

(defn sequence! [ms]! (reduce (mlift2 conj)! (pure [])! ms))


mapM :: Monad m => (a -> m b) -> [a] -> m [b]



(defn mmap! [f vs]! (sequence (map f vs)))



(defn mmap! [f vs]! (sequence (map f vs)))

plus a bunch of others…

Let’s look at a more concrete example

Aggregating movie data(defn cast-by-movie [name]! (future (do (Thread/sleep 5000)! (:cast movie))))!!(defn movies-by-actor [name]! (do (Thread/sleep 2000)! (->> actor-movies! (filter #(= name (:name %)))! first)))!!(defn spouse-of [name]! (do (Thread/sleep 2000)! (->> actor-spouse! (filter #(= name (:name %)))! first)))!!(defn top-5 []! (future (do (Thread/sleep 5000)! top-5-movies)))!

The output

({:name "Cate Blanchett",! :spouse "Andrew Upton",! :movies! ("Lord of The Rings: The Fellowship of The Ring - (top 5)"...)}! {:name "Elijah Wood",! :spouse "Unknown",! :movies! ("Eternal Sunshine of the Spotless Mind"...)}! ...)

One possible solution

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (pmap movies-by-actor @cast)! spouses (pmap spouse-of @cast)! top-5 (top-5)]! (prn "Fetching data...")! (pprint (aggregate-actor-data spouses movies @top-5)))

One possible solution

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (pmap movies-by-actor @cast)! spouses (pmap spouse-of @cast)! top-5 (top-5)]! (prn "Fetching data...")! (pprint (aggregate-actor-data spouses movies @top-5)))

;; "Elapsed time: 10049.334 msecs"

We face the same problems as before

Re-arranging the code would improve it

But the point is not having to do so

In imminent(defn cast-by-movie [name]! (i/future (do (Thread/sleep 5000)! (:cast movie))))!!(defn movies-by-actor [name]! (i/future (do (Thread/sleep 2000)! (->> actor-movies! (filter #(= name (:name %)))! first))))!!(defn spouse-of [name]! (i/future (do (Thread/sleep 2000)! (->> actor-spouse! (filter #(= name (:name %)))! first))))!!(defn top-5 []! (i/future (do (Thread/sleep 5000)! top-5-movies)))

In imminent

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! result (i/sequence [spouses movies (top-5)])]! (prn "Fetching data...")! (pprint (apply aggregate-actor-data! (i/dderef (i/await result)))))

Everything is asynchronous by default

We can optionally block and wait for a result using the “i/await”

function

Ok, that’s cool! But what about Applicative Functors?

Writing this sucks (defn int-f [n]! (i/future (do (Thread/sleep 2000)! (* 2 n))))!!! (-> (i/bind! (int-f 10)! (fn [a] (i/bind! (int-f a)! (fn [b] (i/bind! (int-f b)! (fn [c] ! (i/pure (+ a b c)))))))

Monadic “do” notation

(i/mdo [a (int-f 10)! b (int-f a)! c (int-f b)]! (i/pure (+ a b c)))



How long does this computation take?

What if the computations don’t depend on each other?


(i/mdo [a (int-f 10)! b (int-f 20)! c (int-f 30)]! (i/pure (+ a b c)))



How long does this take now?

The ‘do-notation’ is used to sequence monadic steps so we remain serial, but still asynchronous

Applicative Functor

class Functor f => Applicative f where! pure :: a -> f a! (<*>) :: f (a -> b) -> f a -> f b!

Previous example, rewritten in applicative style

(i/<*> (i/map (int-f 10) (curry + 3))! (int-f 20)! (int-f 30))

;;"Elapsed time: 2001.509 msecs"

It turns out this pattern is a derived function from Applicative Functors

Applicative Functor - derived functions

liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c

Applicative Functor - derived functions

liftA2 :: Applicative f => (a -> b -> c) -> f a -> f b -> f c

(defn alift! [f]! (fn [& as]! {:pre [(seq as)]}! (let [curried (curry f (count as))]! (apply <*>! (fmap curried (first as))! (rest as)))))

“alift” assumes every Applicative Functor is also Functor

Previous example, rewritten using “alift"

;;"Elapsed time: 2001.509 msecs"

((i/alift +) (int-f 10) (int-f 20) (int-f 30))

It looks a lot like function application :)

Applicatives give us the parallel semantics without giving up convenience

Ok, this is awesome! But we have core.async!

The movies example, revisited(defn cast-by-movie [name]! (let [c (chan)]! (go (<! (timeout 5000))! (>! c (:cast movie))! (close! c))! c))!!(defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c))

The movies example, revisited(defn spouse-of [name]! (let [c (chan)]! (go (<! (timeout 2000))! (>! c (->> actor-spouse! (filter #(= name (:name %)))! first))! (close! c))! c))!!(defn top-5 []! (let [c (chan)]! (go (<! (timeout 5000))! (>! c top-5-movies)! (close! c))! c))!!!(defn async-pmap [f source]! (go (->> (map f (<! source))! async/merge! (async/into [])! <!)))

The movies example, revisited

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (aggregate-actor-data (<! spouses)! (<! movies)! (<! top-5))))))!!


“mult” and “tap” have no business being in that code

they are necessary because channels are single-take containers

Exceptionscore.async swallows them by default

(defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c))

Exceptionscore.async swallows them by default

(defn movies-by-actor [name]! (let [c (chan)]! (go (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! c))

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (aggregate-actor-data (<! spouses)! (<! movies)! (<! top-5))))))!!

;; nil - WTF???"

Exceptions - workaround(defn throw-err [e]! (when (instance? Throwable e) (throw e))! e)!!(defmacro <? [ch]! `(throw-err (<! ~ch)))!!!(defn movies-by-actor [name]! (let [c (chan)]! (go (try (do (<! (timeout 2000))! (throw (Exception. (str "Error fetching movies for actor " name)))! (>! c (->> actor-movies! (filter #(= name (:name %)))! first))! (close! c))! (catch Exception e! (do (>! c e)! (close! c)))))! c))

Exceptions - workaround

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! m-cast (mult cast)! movies (async-pmap movies-by-actor (tap m-cast (chan)))! spouses (async-pmap spouse-of (tap m-cast (chan)))! top-5 (top-5)]! (prn "Fetching data...")! (pprint (<!! (go (try! (aggregate-actor-data (<? spouses)! (<? movies)! (<? top-5))! (catch Exception e! e)))))))

A lot of effort for not much benefit

Exceptions - the imminent way *

* not really unique to imminent. Other libraries use this same approach

(defn movies-by-actor [name]! (i/future (do (Thread/sleep 2000)! (throw (Exception. (str "Error fetching movies for actor " name)))! (->> actor-movies! (filter #(= name (:name %)))! first))))

Exceptions - the imminent way # 1

(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! results (i/sequence [spouses movies (top-5)])! _ (prn "Fetching data...")! result (deref (i/await results))]!! (i/map result #(pprint (apply aggregate-actor-data %)))! (i/map-failure result #(pprint (str "Oops: " %)))))

;; Oops: java.lang.Exception: Error fetching movies for actor Cate Blanchett


(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! results (i/sequence [spouses movies (top-5)])! _ (prn "Fetching data...")! result (deref (i/await results))]!! (i/map result #(pprint (apply aggregate-actor-data %)))! (i/map-failure result #(pprint (str "Oops: " %)))))

imminent results are themselves functors :)



(let [cast (cast-by-movie "Lord of The Rings: The Fellowship of The Ring")! movies (i/flatmap cast #(i/map-future movies-by-actor %))! spouses (i/flatmap cast #(i/map-future spouse-of %))! result (i/sequence [spouses movies (top-5)])]! (prn "Fetching data...")! (i/on-success result #(prn-to-repl (apply aggregate-actor-data %)))! (i/on-failure result #(prn-to-repl (str "Oops: " %))))!


Final Thoughts

Category Theory helps you design libraries with higher code reuse due to well known and understood

abstractions

Final Thoughts

Imminent takes advantage of that and provides an asynchronous and composable Futures library for

Clojure

Final Thoughts

Even when compared to core.async, imminent still makes sense unless you need the low level granularity of

channels and/or coordination via queues

Questions?

Leonardo Borges @leonardo_borges

www.leonardoborges.com www.thoughtworks.com

The algebra of library design

Technology

Transcript of The algebra of library design