Clojure Reducers / clj-syd Aug 2012

ReducersA library and model for collection processing in Clojure

Leonardo Borges@leonardo_borgeshttp://www.leonardoborges.comhttp://www.thoughtworks.com

Thursday, 30 August 12

http://www.leonardoborges.com


http://www.thoughtworks.com


ReducersA library and model for collection processing in Clojure


...in 20 mins or le

ss






Reducers huh? Here’s the gist


You get parallel versions of reduce, map and filter





Ta-da! I’m done!




Ta-da! I’m done!

and well under my 20 min limit :)


Alright, alright I’m kidding


How do reducers make parallelism possible?


• JVM’s Fork/Join framework• Reduction Transformers

How do reducers make parallelism possible?


Java requirements

• Fork/Join framework• Java 7 [1] or• Java 6 + the JSR166 jar [2]

Clojure requirements

• 1.5.0-* (this is still MASTER on github [3] as of 30/08/2012)

[1] - http://jdk7.java.net/[2] - http://gee.cs.oswego.edu/dl/jsr166/dist/jsr166.jar[3] - https://github.com/clojure/clojure

Before we start - this is bleeding edge stuff


http://jdk7.java.net/


http://gee.cs.oswego.edu/dl/jsr166/dist/jsr166.jar

http://gee.cs.oswego.edu/dl/jsr166/dist/jsr166.jar

https://github.com/clojure/clojure

https://github.com/clojure/clojure

The Fork/Join Framework


•Based on divide and conquer



•Based on divide and conquer•Work stealing algorithm



•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.



•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold



•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque



•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque•After at least two tasks have finished, results can be combined/joined



•Based on divide and conquer•Work stealing algorithm•Uses deques - double ended queues.•Progressively divides the workload into tasks, up to a threshold•Once it finished one task, it pops another one form its deque•After at least two tasks have finished, results can be combined/joined•Idle workers can pop tasks from the deques of workers which fall behind



Text is boring


Fork/Join algorithm - simplified view



Workload is put in “deques”



...and progressively halved



...up to a configured threshold


Worker 1 Worker 2



Worker 1 Worker 2

Combine



Worker 1 Worker 2



Worker 1 Worker 2

Combine



Worker 1 Worker 2

Combine Combine



Worker 1 Worker 2

Combine



Worker 1 Worker 2



Worker 1 Worker 2

Combine



Worker 1

Combine

Worker 2



Worker 1 Worker 2



Worker 1 Worker 2


Idle workers can “steal” items from other workersThursday, 30 August 12

Worker 1 Worker 2

Combine Combine



Worker 1 Worker 2

Combine



Worker 1 Worker 2


Final result


Let’s talk about Reducers



Motivations

• Performance• via less allocation• via parallelism (leverage Fork/Join)



Motivations

• Performance• via less allocation• via parallelism (leverage Fork/Join)

Issues

• Lists and Seqs are sequential• map / filter implies order


A closer look at what map does

;; a naive map implementation(defn map [f coll] (if (seq coll) (cons (f (first coll)) (map f (rest coll))) '()))



• Recursion




• Recursion• Order




• Recursion• Order• Laziness (not shown)




• Recursion• Order• Laziness (not shown)• Consumes List




• Recursion• Order• Laziness (not shown)• Consumes List• Builds List






Oh, and it also applies the functionto each item before putting the result into the new list





Oh, and it also applies the functionto each item before putting the result into the new list

This is what mapping means!


Reduction Transformers



• Idea is to build map / filter on top of reduce to break from sequentiality



• Idea is to build map / filter on top of reduce to break from sequentiality• map / filter then builds nothing and consumes nothing



• Idea is to build map / filter on top of reduce to break from sequentiality• map / filter then builds nothing and consumes nothing• It changes what reduce means to the collection by transforming the reducing functions


What map is really all about

(defn mapping [f] (fn [f1] (fn [result input] (f1 result (f input)))))


But wait! If map doesn’t consume the list any longer, who does?

• reduce does!• Since Clojure 1.4 reduce lets the collection reduce itself (through the CollReduce / CollFold protocols)• Think of what this means for tree-like structures such as vectors• This is key to leveraging the Fork/Join framework


Now we can use mapping to create reducing functions

(reduce ((mapping inc) +) 0 [1 2 3 4]) ;; 14



(reduce ((mapping inc) +) 0 [1 2 3 4]) ;; 14

(fn [result input] (+ result (inc input)))



(reduce ((mapping inc) conj) [] [1 2 3 4]);; [2 3 4 5]




(fn [result input] (conj result (inc input)))




(fn [result input] (conj result (inc input)))

But it feels awkward to use it in this form


What do we have so far?

• Performance has been improved due to less allocations• No intermediary lists need to be built (see Haskell’s StreamFusion [4])• However reduce is still sequential

[4] - http://bit.ly/streamFusionThursday, 30 August 12



Enters fold


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel• Uses a combining function to join/reduce results


Enters fold

• Takes the sequentiality out or foldl, foldr and reduce• Potentially parallel (fallsback to standard reduce otherwise)• Reduce/Combine strategy (think Fork/Join Framework)• Segments the collection• Runs multiple reduces in parallel• Uses a combining function to join/reduce results

(defn fold [combinef reducef coll] ...)


The combining function is a monoid

• A binary function with an identity element• All the following functions are equivalent monoids




+(+ 2 3) ; 5(+) ; 0




(defn my-+ ([] 0) ([a b] (+ a b)))

(my-+ 2 3) ; 5(my-+) ; 0




(require ‘[clojure.core.reducers :as r])

(def my-+ (r/monoid + (fn [] 0)))

(my-+ 2 3) ; 5(my-+) ; 0


fold by examples

;; all examples assume the reducers library is available as r(ns reducers-playground.core (:require [clojure.core.reducers :as r]))


fold by examples:increment all even positive integers up to 10 million

and add them all up



and add them all up;; these were taken from Rich’s reducers talk



and add them all up;; these were taken from Rich’s reducers talk(def my-vector (into [] (range 10000000)))




(time (reduce + (map inc (filter even? my-vector))))




(time (reduce + (map inc (filter even? my-vector)))) ;; 500msecs





(time (reduce + (r/map inc (r/filter even? my-vector))))





(time (reduce + (r/map inc (r/filter even? my-vector)))) ;; 260msecs






(time (r/fold + (r/map inc (r/filter even? my-vector))))






(time (r/fold + (r/map inc (r/filter even? my-vector)))) ;; 130msecs


fold by examples:standard word count

(def wiki-dump (slurp "subset-wiki-dump50")) ;50 MB

(defn count-words [text] (reduce (fn [memo word] (assoc memo word (inc (get memo word 0)))) {} (map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))


fold by examples:standard word count


(defn count-words [text] (reduce (fn [memo word] (assoc memo word (inc (get memo word 0)))) {} (map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))

(time (count-words wiki-dump)) ;; 45 secs


fold by examples:parallel word count


(defn p-count-words [text] (r/fold (r/monoid (partial merge-with +) hash-map) (fn [memo word] (assoc memo word (inc (get memo word 0)))) (r/map #(.toLowerCase %) (into [] (re-seq #"\w+" text)))))





Combining fn





Will be called at the leaves to merge the partial computations





Will be called with no arguments to provide a seed value



(time (p-count-words wiki-dump)) ;; 30 secs




fold by examples:Load 100k records into PostgreSQL

(def records (into [] (line-seq (BufferedReader. (FileReader. "dump.txt")))))



(time (doseq [record records] (let [tokens (clojure.string/split record #"\t" )] (insert users/users (values { :account-id (nth tokens 0) ... })))))



(time (doseq [record records] (let [tokens (clojure.string/split record #"\t" )] (insert users/users (values { :account-id (nth tokens 0) ... })))))

;; 90 secsThursday, 30 August 12

fold by examples:Load 100k records into PostgreSQL in parallel

(time (r/fold + (r/map (fn [record] (let [tokens (clojure.string/split record #"\t" )] (do (insert users/users (values { :account-id (nth tokens 0) ... })) 1))) records)))


fold by examples:Load 100k records into PostgreSQL in parallel

;; 50 secs

(time (r/fold + (r/map (fn [record] (let [tokens (clojure.string/split record #"\t" )] (do (insert users/users (values { :account-id (nth tokens 0) ... })) 1))) records)))


When to use it


When to use it

• Exploring decision trees


When to use it

• Exploring decision trees• Image processing


When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)


When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)• Basically any list intensive program


When to use it

• Exploring decision trees• Image processing• As a building block for bigger, distributed systems such as Datomic and Cascalog (maybe around parallel agregators)• Basically any list intensive program

But the tools are available to anyone so be creative!


Resources

• The Anatomy of a Reducer - http://bit.ly/anatomyReducers• Rich’s announcement post on Reducers - http://bit.ly/reducersANN• Rich Hickey - Reducers - EuroClojure 2012 - http://bit.ly/reducersVideo (this presentation was heavily inspired by this video)• The Source on github - http://bit.ly/reducersCore



http://bit.ly/reducersANN

http://bit.ly/reducersANN

http://bit.ly/reducersVideo

http://bit.ly/reducersVideo

http://bit.ly/reducersCore

http://bit.ly/reducersCore





Thanks!

Questions?

Leonardo Borges@leonardo_borges

http://www.leonardoborges.comhttp://www.thoughtworks.com






Clojure Reducers / clj-syd Aug 2012

Technology

Transcript of Clojure Reducers / clj-syd Aug 2012