Download - Dataflow: Declarative concurrency in Ruby

Transcript
Page 1: Dataflow: Declarative concurrency in Ruby

DataflowThe declarative concurrent

programming model

Page 2: Dataflow: Declarative concurrency in Ruby

Larry Diehl

{:larrytheliquid => %w[.com github twitter]}

Page 3: Dataflow: Declarative concurrency in Ruby

Outline

Purpose of presentation

Gradual explanation of concepts

Helpful tips

Page 4: Dataflow: Declarative concurrency in Ruby

Purpose

Page 5: Dataflow: Declarative concurrency in Ruby

Lexical Scope

foo = :foodefine_method :foo do fooend

Page 6: Dataflow: Declarative concurrency in Ruby

Dynamic Scope

def foo @fooend

Page 7: Dataflow: Declarative concurrency in Ruby

Mutability

def initialize @foo = :fooend def foo @fooend

Page 8: Dataflow: Declarative concurrency in Ruby

Mutability

def foo @foo = :foo @fooend

Page 9: Dataflow: Declarative concurrency in Ruby

Mutability+Concurrency

def initialize Thread.new { loop { @foo = :shazbot } }end def foo @foo = :foo @fooend

Page 10: Dataflow: Declarative concurrency in Ruby

The Declarative Model

Page 11: Dataflow: Declarative concurrency in Ruby

Declarative Synchronous

my_var = :boundmy_var = :rebind # NOT ALLOWED!

Page 12: Dataflow: Declarative concurrency in Ruby

Declarative Synchronous

local do |my_var| my_var.object_id # thread sleepsend

Page 13: Dataflow: Declarative concurrency in Ruby

Declarative Synchronous

local do |my_var| unify my_var, :bound unify my_var, :rebind # => # Dataflow::UnificationError, # ":bound != :rebind"end

Page 14: Dataflow: Declarative concurrency in Ruby

Declarative Synchronous

class MyClass declare :my_var def initialize unify my_var, :bound endend

Page 15: Dataflow: Declarative concurrency in Ruby

Declarative Concurrent(MAGIC)

Page 16: Dataflow: Declarative concurrency in Ruby

Declarative Concurrent

local do |my_var| Thread.new { unify my_var, :bound } my_var.should == :boundend

Page 17: Dataflow: Declarative concurrency in Ruby

Dependency Resolution

local do |sentence, middle, tail| Thread.new { unify middle, "base are belong #{tail}" } Thread.new { unify tail, "to us" } Thread.new { unify sentence, "all your #{middle}" } sentence.should == "all your base are belong to us"end

Page 18: Dataflow: Declarative concurrency in Ruby

Asynchronous Outputdef Worker.async(output=nil) Thread.new do result = # do hard work unify output, result if output endend local do |output| Worker.async(output) output.should == # hard work resultend

Page 19: Dataflow: Declarative concurrency in Ruby

Asynchronous Output

local do |output| flow(output) do # do hard work end output.should == # hard work resultend

Page 20: Dataflow: Declarative concurrency in Ruby

Anonymous variables

{'google.com' => Dataflow::Variable.new, 'bing.com' => Dataflow::Variable.new}.map do |domain,var| Thread.new do unify var, open("http://#{domain}").read end varend

Page 21: Dataflow: Declarative concurrency in Ruby

need_later

%w[google.com bing.com].map do |domain| need_later { open("http://#{domain}").read }end

Page 22: Dataflow: Declarative concurrency in Ruby

Chunked Sequential Processing

(1..100).each_slice(10).map do |chunk| sleep(1) chunk.inject(&:+)end.inject(&:+) # => ~10s

Page 23: Dataflow: Declarative concurrency in Ruby

Chunked Parallel Processing

(1..100).each_slice(10).map do |chunk| need_later do sleep(1) chunk.inject(&:+) endend.inject(&:+) # => ~1s

Page 24: Dataflow: Declarative concurrency in Ruby

Leaving Declarative via Async

Page 25: Dataflow: Declarative concurrency in Ruby

Ports & Streams

local do |port, stream| unify port, Dataflow::Port.new(stream) port.send 1 port.send 2 stream.take(2).should == [1, 2]end

Page 26: Dataflow: Declarative concurrency in Ruby

Ports & Streams (async)local do |port, stream| unify port, Dataflow::Port.new(stream) Thread.new do stream.each do |message| puts "received: #{message}" end end %w[x y z].each do |letter| Thread.new{ port.send letter } end stream.take(3).sort.should == %w[x y z]end

Page 27: Dataflow: Declarative concurrency in Ruby

FutureQueuelocal do |queue, first, second, third| unify queue, FutureQueue.new queue.pop first queue.pop second queue.push 1 queue.push 2 queue.push 3 queue.pop third [first, second, third].should == [1, 2, 3]end

Page 28: Dataflow: Declarative concurrency in Ruby

ActorsPing = Actor.new { 3.times { case receive when :ping puts "Ping" Pong.send :pong end }}

Pong = Actor.new { 3.times { case receive when :pong puts "Pong" Ping.send :ping end }}

Ping.send :ping

Page 29: Dataflow: Declarative concurrency in Ruby

by_need

def baz(num) might_get_used = by_need { Factory.gen } might_get_used.value if num%2 == 0end

Page 30: Dataflow: Declarative concurrency in Ruby

Tips

Page 31: Dataflow: Declarative concurrency in Ruby

Modular

local do |my_var| Thread.new { unify my_var, :bound } # my_var.wait my_var.should == :boundend

Page 32: Dataflow: Declarative concurrency in Ruby

Debugging

local do |my_var| my_var.inspect # => #<Dataflow::Variable:2637860 unbound>end

Page 33: Dataflow: Declarative concurrency in Ruby

Class/Module methods

Dataflow.local do |my_var| Dataflow.async do Dataflow.unify my_var, :bound end my_var.should == :boundend

Page 34: Dataflow: Declarative concurrency in Ruby

Use Casesgeneral purpose

concurrency for elegant program structure with respect to coordination

concurrency to make use of extra processors/cores (depending on Ruby implementation)

web developmentworker daemons

concurrently munging together data from various rest api's

Page 35: Dataflow: Declarative concurrency in Ruby

Ruby Implementations

Pure Ruby library, should work on any implementation

JRuby in particular has a great GC, no GIL, native threads, and a tunable threadpool option.

Rubinius has more code written in Ruby, so it proxies more method calls (e.g. Array#flatten).

Page 36: Dataflow: Declarative concurrency in Ruby

class FutureQueue include Dataflow declare :push_port, :pop_port def initialize local do |pushed, popped| unify push_port, Dataflow::Port.new(pushed) unify pop_port, Dataflow::Port.new(popped) Thread.new { loop do barrier pushed.head, popped.head unify popped.head, pushed.head pushed, popped = pushed.tail, popped.tail end } end end def push(x) push_port.send x end def pop(x) pop_port.send x endend

Page 37: Dataflow: Declarative concurrency in Ruby

The End

sudo port install dataflow

http://github.com/larrytheliquid/dataflow

freenode: #dataflow-gem