Dataflow: Declarative concurrency in Ruby
-
Upload
larry-diehl -
Category
Technology
-
view
2.570 -
download
1
description
Transcript of Dataflow: Declarative concurrency in Ruby
DataflowThe declarative concurrent
programming model
Larry Diehl
{:larrytheliquid => %w[.com github twitter]}
Outline
Purpose of presentation
Gradual explanation of concepts
Helpful tips
Purpose
Lexical Scope
foo = :foodefine_method :foo do fooend
Dynamic Scope
def foo @fooend
Mutability
def initialize @foo = :fooend def foo @fooend
Mutability
def foo @foo = :foo @fooend
Mutability+Concurrency
def initialize Thread.new { loop { @foo = :shazbot } }end def foo @foo = :foo @fooend
The Declarative Model
Declarative Synchronous
my_var = :boundmy_var = :rebind # NOT ALLOWED!
Declarative Synchronous
local do |my_var| my_var.object_id # thread sleepsend
Declarative Synchronous
local do |my_var| unify my_var, :bound unify my_var, :rebind # => # Dataflow::UnificationError, # ":bound != :rebind"end
Declarative Synchronous
class MyClass declare :my_var def initialize unify my_var, :bound endend
Declarative Concurrent(MAGIC)
Declarative Concurrent
local do |my_var| Thread.new { unify my_var, :bound } my_var.should == :boundend
Dependency Resolution
local do |sentence, middle, tail| Thread.new { unify middle, "base are belong #{tail}" } Thread.new { unify tail, "to us" } Thread.new { unify sentence, "all your #{middle}" } sentence.should == "all your base are belong to us"end
Asynchronous Outputdef Worker.async(output=nil) Thread.new do result = # do hard work unify output, result if output endend local do |output| Worker.async(output) output.should == # hard work resultend
Asynchronous Output
local do |output| flow(output) do # do hard work end output.should == # hard work resultend
Anonymous variables
{'google.com' => Dataflow::Variable.new, 'bing.com' => Dataflow::Variable.new}.map do |domain,var| Thread.new do unify var, open("http://#{domain}").read end varend
need_later
%w[google.com bing.com].map do |domain| need_later { open("http://#{domain}").read }end
Chunked Sequential Processing
(1..100).each_slice(10).map do |chunk| sleep(1) chunk.inject(&:+)end.inject(&:+) # => ~10s
Chunked Parallel Processing
(1..100).each_slice(10).map do |chunk| need_later do sleep(1) chunk.inject(&:+) endend.inject(&:+) # => ~1s
Leaving Declarative via Async
Ports & Streams
local do |port, stream| unify port, Dataflow::Port.new(stream) port.send 1 port.send 2 stream.take(2).should == [1, 2]end
Ports & Streams (async)local do |port, stream| unify port, Dataflow::Port.new(stream) Thread.new do stream.each do |message| puts "received: #{message}" end end %w[x y z].each do |letter| Thread.new{ port.send letter } end stream.take(3).sort.should == %w[x y z]end
FutureQueuelocal do |queue, first, second, third| unify queue, FutureQueue.new queue.pop first queue.pop second queue.push 1 queue.push 2 queue.push 3 queue.pop third [first, second, third].should == [1, 2, 3]end
ActorsPing = Actor.new { 3.times { case receive when :ping puts "Ping" Pong.send :pong end }}
Pong = Actor.new { 3.times { case receive when :pong puts "Pong" Ping.send :ping end }}
Ping.send :ping
by_need
def baz(num) might_get_used = by_need { Factory.gen } might_get_used.value if num%2 == 0end
Tips
Modular
local do |my_var| Thread.new { unify my_var, :bound } # my_var.wait my_var.should == :boundend
Debugging
local do |my_var| my_var.inspect # => #<Dataflow::Variable:2637860 unbound>end
Class/Module methods
Dataflow.local do |my_var| Dataflow.async do Dataflow.unify my_var, :bound end my_var.should == :boundend
Use Casesgeneral purpose
concurrency for elegant program structure with respect to coordination
concurrency to make use of extra processors/cores (depending on Ruby implementation)
web developmentworker daemons
concurrently munging together data from various rest api's
Ruby Implementations
Pure Ruby library, should work on any implementation
JRuby in particular has a great GC, no GIL, native threads, and a tunable threadpool option.
Rubinius has more code written in Ruby, so it proxies more method calls (e.g. Array#flatten).
class FutureQueue include Dataflow declare :push_port, :pop_port def initialize local do |pushed, popped| unify push_port, Dataflow::Port.new(pushed) unify pop_port, Dataflow::Port.new(popped) Thread.new { loop do barrier pushed.head, popped.head unify popped.head, pushed.head pushed, popped = pushed.tail, popped.tail end } end end def push(x) push_port.send x end def pop(x) pop_port.send x endend
The End
sudo port install dataflow
http://github.com/larrytheliquid/dataflow
freenode: #dataflow-gem