Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time Ruby for the Real-Time Webaka IM for web-applications
Ilya GrigorikCTO / PostRank
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
www.postrank.com www.igvita.com@igrigorik
Background: - PHP, Perl - Ruby + Rails from ‘06
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
The slides… Questions & Comments My blog
• Real-Time: the hype & the technology• Real-Time: the benefits
• XMPP• AMQP• PSHB• WebHooks
• Ruby examples• Real-life applications
Fully buzzword compliant!
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
The Hype! Make it stop!
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time has many definitionsIt all depends on your context
(micro/nano) seconds“Solution Exhibits Under 700 Nanoseconds of Latency for Inter-Process Communication Messaging”
milliseconds
seconds
minutes / hours
500ms is real-time enough to feel real-time for IM.
Real-time web is IM for web-services
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
+ New Applications
+ Better Architecture
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Polling: painful, wasteful
Data? No Data? No Data? Yes
Many wasteful checks
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Extensible Messaging and Presence Protocol
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Extensible Messaging and Presence Protocol (XMPP)
From: A, To: BHello!
From: A, To: BHello!
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP Features
Persistent connections
Event-stream protocol
Identity and authentication
Presence
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
JID: Federation, Identity & Authentication
User
Jabber Software Foundation
[email protected]/office
Domain Resource
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Example: Message Routing with XMPP
<message from="[email protected]/office" type="chat" to="[email protected]" id="aae1a"> <body>hello</body> <active xmlns="http://jabber.org/protocol/chatstates"/></message>
Verbose protocol (XML)
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP in the wild: Google Talk
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP in the wild: Google Talk + Video
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Psi: cross-platform Jabber/XMPP client
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP4R (Ruby) Demo
require "rubygems"require "xmpp4r"
jid = Jabber::JID::new("[email protected]")client = Jabber::Client.new(jid)client.connect("talk.google.com")client.auth("password")
to = "[email protected]"subject = "Jabber client"message = "Hello XMPP World!"
piclient.send Jabber::Message::new(to, message).set_subject(subject)
# <message to='[email protected]'># <body>Hello XMPP World!</body># <subject>Jabber client</subject># </message>
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP4R (Ruby) Demo
require "rubygems"require "xmpp4r"
jid = Jabber::JID::new("[email protected]")client = Jabber::Client.new(jid)client.connect("talk.google.com")client.auth("password")
to = "[email protected]"subject = "Jabber client"message = "Hello XMPP World!"
piclient.send Jabber::Message::new(to, message).set_subject(subject)
# <message to='[email protected]'># <body>Hello XMPP World!</body># <subject>Jabber client</subject># </message>
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP4R (Ruby) Demo
client.send(Jabber::Presence.new.set_type(:away))
# <presence from='[email protected]/iMac8D2CB97D' to='[email protected]/0EDD826C' xmlns='jabber:client'># <show>away</show># <priority>0</priority># <x xmlns='http://www.apple.com/xmpp/idle'># <idle-since>2009-04-01T21:48:15Z</idle-since># </x># </presence>
client.add_message_callback do |m| puts "#{m.from} -- #{m.body}"end
# > [email protected] -- Hey!
Client Idle…
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP4R (Ruby) Demo
client.send(Jabber::Presence.new.set_type(:available))
# <presence from='[email protected]/iMac8D2CB97D' to='[email protected]/0EDD826C' xmlns='jabber:client'># <show>away</show># <priority>0</priority># <x xmlns='http://www.apple.com/xmpp/idle'># <idle-since>2009-04-01T21:48:15Z</idle-since># </x># </presence>
client.add_message_callback do |m| puts "#{m.from} -- #{m.body}"end
# > [email protected] -- Hey!
Client Idle…
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
One-to-many distribution + C2S
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XEP-0060: Publish-Subscribe (Pubsub)
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Publish-Subscribe
Subscribe New message!
Persistent connection
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
<iq type='set‘ from='[email protected]/blogbot' to='pubsub.shakespeare.lit' id='pub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <publish node='princely_musings'>
<item> <entry xmlns='http://www.w3.org/2005/Atom'> <title>Soliloquy<title> <summary> To be, or not to be: that is the question! <summary> <link rel='alternate' type='text/html' href='http://denmark.lit/2003/12/13/atom03'/> <id>tag:denmark.lit,2003:entry-32397</id> <published>2003-12-13T18:30:02Z</published> <updated>2003-12-13T18:30:02Z</updated> </entry> </item>
</publish> </pubsub></iq>
IQ Stanza
PubSub Protocol: Client XML
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
<iq type='set‘ from='[email protected]/blogbot' to='pubsub.shakespeare.lit' id='pub1'> <pubsub xmlns='http://jabber.org/protocol/pubsub'> <publish node='princely_musings'>
<item> <entry xmlns='http://www.w3.org/2005/Atom'> <title>Soliloquy<title> <summary> To be, or not to be: that is the question! <summary> <link rel='alternate' type='text/html' href='http://denmark.lit/2003/12/13/atom03'/> <id>tag:denmark.lit,2003:entry-32397</id> <published>2003-12-13T18:30:02Z</published> <updated>2003-12-13T18:30:02Z</updated> </entry> </item>
</publish> </pubsub></iq>
AtomPub
PubSub Protocol: Client XML
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XEP-0060: Publish-Subscribe (Pubsub)Publish
Distribute
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
<message from='pubsub.shakespeare.lit' to='[email protected]' id='foo'> <event xmlns='http://jabber.org/protocol/pubsub#event'> <items node='princely_musings'> <item id='ae890ac52d0df67ed7cfdf51b644e901'> [ ... ENTRY ... ] </item> </items> </event></message>
<message from='pubsub.shakespeare.lit' to='[email protected]' id='bar'> <event xmlns='http://jabber.org/protocol/pubsub#event'> <items node='princely_musings'> <item id='ae890ac52d0df67ed7cfdf51b644e901'> [ ... ENTRY ... ] </item> </items> </event></message>
Subscriber A
XEP-0060: Publish-Subscribe (Pubsub)
Subscriber B
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-time communication
XMPP XMPPXMPP
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Ruby + FireEagle via XMPP
require 'fire_hydrant'require 'yaml'
hydrant = FireHydrant.new(YAML.load(File.read("config.yml")))
hydrant.on_location_update do |user| puts "#{user.token} has moved to #{user.locations.first}."end
hydrant.run!
Push notifications
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
XMPP / Jabber Servers
Ejabberd ErlangDjabberd PerlOpenFire JavaTigase Java RPM, GUI, shiny
Defacto XMPP server
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Advanced Message Queuing Protocol (AMQP)
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Working Group (16 companies)
“AMQP is an open Internet Protocol for Business Messaging”
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP ArchitecturePublisher
AMQP Broker Consumer
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Clustering
Broker Clustering
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Broker Internals
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.amzMessage: I like AMZ!
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Direct Exchange
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.amzMessage: I like AMZ!
Queue Name: amazonBind: usd.stock.amz
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Topic Exchange
Direct Exchange
Topic Exchange
Fanout Exchange
Queue Name: stocksBind: usd.stock.*
Routing key: usd.stock.amzMessage: I like AMZ!
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Topic Exchange
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.msftMessage: I like Microsoft!
Message: I like Microsoft! Queue
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Fanout Exchange
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.msftMessage: I like Microsoft!
Queue 2
Name: stocksBind: “”
Queue 1
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Fanout Exchange
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.msftMessage: I like Microsoft!
Queue 2 Message: I like MicrosoftQueue
1
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Kung-fu: Load-balancing
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
AMQP Load Balancing
Direct Exchange
Topic Exchange
Fanout Exchange
Routing key: usd.stock.amzMessage: I like AMZ!
Only one client gets the message!Queue
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Elastic AMQP Load-balancing
GET /purchase
OK
Bind: purchase.pdf
HAProxy
App server BApp server A
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
More AMQP Kung-fu: - Pubsub - Key routing - Failover - Instant feedback - At least once, exactly-once - …
http://bit.ly/igvita-amqp
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
require 'mq' AMQP.start(:host => 'amqp-server.com') do
mq = MQ.new mq.topic('stocks').publish("5.95", :key => "usd.amz") end
AMQP.start(:host => 'amqp-server.com') do mq = MQ.new mq.queue(’stocks').bind(mq.topic('stocks'), :key => 'usd.*').subscribe{ |price| print ’stock quote: ', price } end
Publisher
AMQP + Ruby Example
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
require 'mq' AMQP.start(:host => 'amqp-server.com') do
mq = MQ.new mq.topic('stocks').publish("5.95", :key => "usd.amz") end
AMQP.start(:host => 'amqp-server.com') do mq = MQ.new mq.queue(‘stocks').bind(mq.topic('stocks'), :key => 'usd.*').subscribe{ |price| print ‘stock quote: ', price } end
Consumer
AMQP + Ruby Example
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
http://blog.webhooks.org/2009/04/23/slides-from-pivotal-labs-talk/
WebHooks:Pattern for enabling user-defined callbacks in web-applications
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
http://blog.webhooks.org/2009/04/23/slides-from-pivotal-labs-talk/
WebHooks @ PayPal
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
http://blog.webhooks.org/2009/04/23/slides-from-pivotal-labs-talk/
WebHooks Workflow
HAProxy
App server BApp server A
1
/register http://callback-1.com/register http://callback-2.com
ok
http://blog.webhooks.org/2009/04/23/slides-from-pivotal-labs-talk/
HAProxy
App server BApp server A
2
/post Hello World!
ok
3/post Hello World!
/post Hello World!
http://callback-1.com
http://callback-2.com
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
WebHooks WorkflowWebHooks @ GitHub
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Rails ActiveRecord + WebHooks
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Watercoolr: Ruby WebHooks Servervia a simple Sinatra app
http://github.com/jcapote/watercoolr
→ POST /channels← { 'id':'2d0128d' }1
2 → POST /subscribers data={ 'channel':'2d0128d', 'url':'http://api.calback.com/handler' }← { 'status': 'OK' }
you → POST /messages data={ 'channel':'2d0128d', 'message':'hey guys!' } watercoolr → POST http://api.callback.com/handler data='hey guys!' ...for every subscriber...← { 'status': 'OK' }
3
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Watercoolr on Google App EngineDataMapper + Sinatra + Jruby
require 'rubygems'require 'rest_client'require 'json'
# ruby postbin.rb http://www.postbin.org/1j11vyp
puts "creating channel..."resp = RestClient.post 'http://watercoolr.appspot.com/channels', :data => ''id = JSON.parse(resp)["id"]
puts "adding subscriber to channel #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/subscribers', :data => { :channel => id, :url => ARGV[0] }.to_json
puts resp # {"status":"OK"}
puts "posting message to #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/messages', :data => { :channel => id, :message => 'Hello World' }.to_json
puts resp # {"status":"OK"}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Watercoolr on Google App EngineDataMapper + Sinatra + Jruby
require 'rubygems'require 'rest_client'require 'json'
# ruby postbin.rb http://www.postbin.org/1j11vyp
puts "creating channel..."resp = RestClient.post 'http://watercoolr.appspot.com/channels', :data => ''id = JSON.parse(resp)["id"]
puts "adding subscriber to channel #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/subscribers', :data => { :channel => id, :url => ARGV[0] }.to_json
puts resp # {"status":"OK"}
puts "posting message to #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/messages', :data => { :channel => id, :message => 'Hello World' }.to_json
puts resp # {"status":"OK"}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Watercoolr on Google App EngineDataMapper + Sinatra + Jruby
require 'rubygems'require 'rest_client'require 'json'
# ruby postbin.rb http://www.postbin.org/1j11vyp
puts "creating channel..."resp = RestClient.post 'http://watercoolr.appspot.com/channels', :data => ''id = JSON.parse(resp)["id"]
puts "adding subscriber to channel #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/subscribers', :data => { :channel => id, :url => ARGV[0] }.to_json
puts resp # {"status":"OK"}
puts "posting message to #{id}"resp = RestClient.post 'http://watercoolr.appspot.com/messages', :data => { :channel => id, :message => 'Hello World' }.to_json
puts resp # {"status":"OK"}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
http://bit.ly/igvita-watercoolrhttp://www.github.com/igrigorik/watercoolrhttp://www.postbin.org
POST’ing to PostBingreat debugging tool
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
PubSub over HTTPbasically, WebHooks…
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
“A simple, open, server-to-server web-hook-based pubsub (publish/subscribe) protocol as an extension to Atom and RSS.”
“Parties (servers) speaking the PubSubHubbub protocol can get near-instant notifications (via webhook callbacks) when a topic (feed URL) they're interested in is updated.”
http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.2.html
+ Spec’ed PubSub Protocol
+ Deployed & Available
+ XML Transport
- XML Transport
- Not as general purpose
- No firehose
http://docs.google.com/present/view?id=ajd8t6gk4mh2_34dvbpchfs
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
1
2
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
3
4
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
PSHB in the WildGoogle, Typepad, Wordpress…
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Consuming PSHB in Rubygem install pubsubhubbub
require 'pubsubhubbub'
EventMachine.run { hub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/') # subscribe to real-time notifications hub.subscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” # unsubscribe hub.unsubscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” hub.callback { puts "Processed" } hub.errback { puts "Error! " + hub.response_header.status.to_s }}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Consuming PSHB in Rubygem install pubsubhubbub
require 'pubsubhubbub'
EventMachine.run { hub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/') # subscribe to real-time notifications hub.subscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” # unsubscribe hub.unsubscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” hub.callback { puts "Processed" } hub.errback { puts "Error! " + hub.response_header.status.to_s }}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Consuming PSHB in Rubygem install pubsubhubbub
require 'pubsubhubbub'
EventMachine.run { hub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/') # subscribe to real-time notifications hub.subscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” # unsubscribe hub.unsubscribe "http://blog.yoursite.com/atom.xml", "http://yourhub.com/hubbub” hub.callback { puts "Processed" } hub.errback { puts "Error! " + hub.response_header.status.to_s }}
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Publishing PSHB in Rubygem install pubsubhubbub
require 'pubsubhubbub'
EventMachine.run { hub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish') # publish notification to all subscribers hub.publish ['http://www.test.com', 'http://www.test.com/2'] hub.callback { puts "Notified PSHB Hub" } hub.errback { puts "Notification failed" + hub.response_header.status.to_s } }
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Publishing PSHB in Rubygem install pubsubhubbub
require 'pubsubhubbub'
EventMachine.run { hub = EventMachine::PubSubHubbub.new('http://pubsubhubbub.appspot.com/publish') # publish notification to all subscribers hub.publish ['http://www.test.com', 'http://www.test.com/2'] hub.callback { puts "Notified PSHB Hub" } hub.errback { puts "Notification failed" + hub.response_header.status.to_s } }
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
http://github.com/igrigorik/pubsubhubbub
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Real-Time Web = “IM for web applications”
• XMPP : Presence• AMQP : Routing
• WebHooks : Extensibility• PubsubHubbub : PubSub over HTTP
Real-Time Ruby & Real-Time Web @igrigorik #railssummit #realtimehttp://bit.ly/eda
Questions?
The slides… Questions & Comments My blog
http://bit.ly/igvita-amqphttp://bit.ly/igvita-webhookshttp://bit.ly/igvita-pshb
Related Blog Posts:
Top Related