Asynchronous Processing with Ruby on Rails (RailsConf 2008)
-
Upload
jonathan-dahl -
Category
Technology
-
view
22.879 -
download
0
description
Transcript of Asynchronous Processing with Ruby on Rails (RailsConf 2008)
![Page 1: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/1.jpg)
Asynchronous Processing
Jonathan Dahl
(and RailSpikes,Slantwise,Zencoder,
etc.)
![Page 2: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/2.jpg)
I. What is it, and
why should I care?
![Page 3: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/3.jpg)
![Page 4: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/4.jpg)
Wife: What are you talking about at RailsConf this year?
Jon:
Wife:
Asynchronous Processing
[changes subject]
![Page 5: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/5.jpg)
important tool
![Page 6: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/6.jpg)
(ahem...)
![Page 7: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/7.jpg)
![Page 8: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/8.jpg)
Related Concepts
• Background Processing
• Parallel Processing
• Distributed Processing
![Page 9: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/9.jpg)
has_attachment :storage => :s3
![Page 10: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/10.jpg)
Browser Response
Send to S3
Image Upload ( 15 seconds)
![Page 11: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/11.jpg)
( 15 seconds)
Browser Response
Send to S3
Image Upload
![Page 12: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/12.jpg)
Browser Response
Send to S3
Image Upload
( 3 seconds)
![Page 13: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/13.jpg)
Browser Response
Send to S3
Image Upload
![Page 14: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/14.jpg)
has_attachment :storage => :file_system
![Page 15: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/15.jpg)
Send to S3
Browser Response
Image Upload ( 15 seconds)
![Page 16: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/16.jpg)
Send to S3
Browser Response
Image Upload
( 3 seconds)
( who cares?)
![Page 17: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/17.jpg)
has_attachment :storage => :s3, :thumbnails => { :thumb => '100x100!', :small => '240x180>', :medium => '500x500>' }
![Page 18: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/18.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 19: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/19.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 20: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/20.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 21: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/21.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 22: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/22.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 23: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/23.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 24: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/24.jpg)
Browser Response
Send Thumbnail A to S3
Image Upload
Generate 3 Thumbnails
Send Thumbnail B to S3
Send Thumbnail C to S3
Send Original to S3
![Page 25: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/25.jpg)
![Page 26: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/26.jpg)
II. When do I need it?
![Page 27: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/27.jpg)
Time
![Page 28: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/28.jpg)
![Page 29: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/29.jpg)
Request
• Method (GET, POST)
• URI (host, port, path)
• Parameters
![Page 30: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/30.jpg)
Response
• Status (200, 404, 500)
• Metadata (content type, server info, etc.)
• Body (xml, html, file)
![Page 31: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/31.jpg)
Resources
![Page 32: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/32.jpg)
Trigger
![Page 33: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/33.jpg)
HTTP trigger - browser request
![Page 34: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/34.jpg)
GET /photos/1.xml HTTP/1.1Host: example.com:80
HTTP trigger - API request
![Page 35: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/35.jpg)
cap staging deploy
Human trigger - capistrano
![Page 36: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/36.jpg)
rake db:migrate
Human trigger - rake
![Page 37: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/37.jpg)
$ script/console productionLoading production environment (Rails 2.0.2)>> Photo.destroy_all
Human trigger - console
![Page 38: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/38.jpg)
- Send email in 2 hours
- Sync data at 3:00am PST
- Notify admin when disk is 90% full
- Expire sessions that are inactive
- Archive records that exceed quota
No trigger?
![Page 39: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/39.jpg)
1. Time2. Resources3. Trigger
![Page 40: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/40.jpg)
Concrete examples
• Sending mail
• Transcoding video/audio
• Storing images on S3
• Receiving email
• Synching with outside database
• Complex computations
![Page 41: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/41.jpg)
![Page 42: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/42.jpg)
class Emailer < ActionMailer::ARMailer
![Page 43: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/43.jpg)
![Page 44: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/44.jpg)
![Page 45: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/45.jpg)
ZencoderUser
Zencoder Manager
Worker
Video SharingWebsite
Data Storage(Amazon S3)
Worker Worker
Worker
![Page 46: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/46.jpg)
![Page 47: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/47.jpg)
class Photo < ActiveRecord::Base after_create :background_s3_upload def background_s3_upload Bj.submit "./script/runner ./jobs/send_to_s3.rb #{self.id}" endend
![Page 48: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/48.jpg)
![Page 49: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/49.jpg)
![Page 50: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/50.jpg)
![Page 51: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/51.jpg)
III. So how do you
decide what to use?
![Page 52: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/52.jpg)
be seamless
![Page 53: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/53.jpg)
how reliable?
![Page 54: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/54.jpg)
when should it run?
![Page 55: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/55.jpg)
dependencies and system complexity
![Page 56: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/56.jpg)
scaling and/or
performance
![Page 57: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/57.jpg)
IV. The simple solution:
fork or thread
![Page 58: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/58.jpg)
![Page 59: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/59.jpg)
![Page 60: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/60.jpg)
![Page 61: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/61.jpg)
Parallel vs. Background
![Page 62: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/62.jpg)
1. Stay within one request
![Page 63: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/63.jpg)
2. thread.join
![Page 64: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/64.jpg)
3. ActiveRecordActiveRecord::Base.allow_concurrency = true
![Page 65: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/65.jpg)
![Page 66: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/66.jpg)
fire and forget
![Page 67: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/67.jpg)
Spawn
spawn(:method => :fork) do # do somethingend
![Page 68: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/68.jpg)
1. Time2. Resources
3. Trigger
![Page 69: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/69.jpg)
![Page 70: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/70.jpg)
V. More robust
solutions
![Page 71: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/71.jpg)
Task StorageTask Trigger
![Page 72: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/72.jpg)
Task Storage
• task details (what happens?)
• priority
• when to run
![Page 73: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/73.jpg)
Task Trigger
• worker pulling jobs
• time-based
• execute immediately
![Page 74: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/74.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 75: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/75.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 76: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/76.jpg)
create_table "jobs" do |t| t.text "command" t.integer "priority" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.text "result"end
![Page 77: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/77.jpg)
create_table "photos" do |t| t.string "filename" t.datetime "created_at" t.datetime "processed_at"endt.datetime "processed_at"
![Page 78: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/78.jpg)
create_table "photos" do |t| t.string "filename" t.datetime "created_at" t.datetime "processed_at"endt.datetime "processed_at"
![Page 79: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/79.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 80: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/80.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 81: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/81.jpg)
• Amazon SQS
• Websphere MQ
• Starling
• JMS
• beanstalkd
![Page 82: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/82.jpg)
queue = SQS.get_queue("task_list")
![Page 83: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/83.jpg)
queue.send_message "process:2872"
put message
![Page 84: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/84.jpg)
message = queue.receive_message
receive message
![Page 85: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/85.jpg)
Starling
starling -h 192.168.1.1 -d
![Page 86: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/86.jpg)
require 'memcache'starling = MemCache.new('192.168.1.1:22122')
# Put messages onto a queue:starling.set('my_queue', 12345)
# Get message from the queue:starling.get('my_queue')
![Page 87: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/87.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 88: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/88.jpg)
storage choice?
• queue: optimized for performance
• database: you’ve already got one
![Page 89: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/89.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 90: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/90.jpg)
daemon
#!/usr/bin/env rubyclass JobRequester < SimpleDaemon::Base def self.start loop { Job.process_next } endend
JobRequester.daemonize
![Page 91: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/91.jpg)
![Page 92: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/92.jpg)
Task StorageDatabase
Message Queue
Task Triggerdaemon
cron
![Page 93: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/93.jpg)
0 6 * * * script/runner jobs/send_emails.rb
![Page 94: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/94.jpg)
cronedit
require 'cronedit'
CronEdit::Crontab.Add "send-emails", { :minute => 0, :hour => 6, :command => "script/runner jobs/send_emails.rb" }
CronEdit::Crontab.Remove 'old-task'
![Page 95: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/95.jpg)
trigger choice
• process: always running
• cron: as reliable as your operating system
![Page 96: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/96.jpg)
![Page 97: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/97.jpg)
BackgroundDRbclass BillingWorker < BackgrounDRb::MetaWorker set_worker_name :billing_worker def create(args = nil) # this method is called when worker is loaded for the first time end
def charge_customer(customer_id = nil) logger.info 'charging customer now' endend
MiddleMan.worker(:billing_worker).charge_customer(current_customer.id)
![Page 98: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/98.jpg)
:backgroundrb: :ip: 0.0.0.0
:development: :backgroundrb: :port: 11111 :log: foreground
:production: :backgroundrb: :port: 22222 :lazy_load: true :debug_log: false ./script/backgroundrb start
![Page 99: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/99.jpg)
AP4Rdef MyController def queue ap4r.async_to({:action => 'download'}, {:story => story.id, :url => params[:url]}) end
def download # long-running task endend
![Page 100: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/100.jpg)
Bj
Acronym
![Page 101: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/101.jpg)
create_table "bj_job", :primary_key => "bj_job_id", :force => true do |t| t.text "command" t.text "state" t.integer "priority" t.text "tag" t.integer "is_restartable" t.text "submitter" t.text "runner" t.integer "pid" t.datetime "submitted_at" t.datetime "started_at" t.datetime "finished_at" t.text "env" t.text "stdin" t.text "stdout" t.text "stderr" t.integer "exit_status"end
![Page 102: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/102.jpg)
Bj.submit "./script/runner ./jobs/task.rb"
![Page 103: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/103.jpg)
after_create :bj_send_to_s3
def bj_send_to_s3 Bj.submit "./script/runner ./jobs/send.rb #{id}"end
![Page 104: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/104.jpg)
![Page 105: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/105.jpg)
# environment configWorkling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
Workling
![Page 106: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/106.jpg)
# task classclass ImageWorker < Workling::Base def send_to_s3(options = {}) # put file to S3 endend
![Page 107: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/107.jpg)
# trigger asynchronous jobImageWorker.asynch_send_to_s3(:image_id => 2927)
![Page 108: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/108.jpg)
script/workling_starling_client start
![Page 109: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/109.jpg)
Pitfalls
![Page 110: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/110.jpg)
race conditions
![Page 111: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/111.jpg)
alive, but stalled
![Page 112: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/112.jpg)
VI.some
recommendations
![Page 113: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/113.jpg)
general purpose
Bj
![Page 114: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/114.jpg)
distributed processing
SQS(+ custom worker)
![Page 115: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/115.jpg)
time-scheduled
cron(+ rake or script)
![Page 116: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/116.jpg)
speed + scalability
Starling/Workling
![Page 117: Asynchronous Processing with Ruby on Rails (RailsConf 2008)](https://reader034.fdocuments.us/reader034/viewer/2022052609/55493936b4c905194d8b4a5a/html5/thumbnails/117.jpg)
Thanks!Jonathan Dahl
Slides at RailSpikes http://railspikes.com