Social Real Time Apps Pub Sub Hubbub
-
Upload
julien-genestoux -
Category
Documents
-
view
113 -
download
1
Transcript of Social Real Time Apps Pub Sub Hubbub
![Page 1: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/1.jpg)
![Page 2: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/2.jpg)
Make your application real-timewith PubSubHubbubBrett SlatkinMay 19th, 2010
![Page 3: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/3.jpg)
View live notes and ask questions about this session on Google Wave
http://tinyurl.com/push-io2010
Mehttp://onebigfluke.com
![Page 4: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/4.jpg)
Agenda
IntroPublishingSubscribingHubs Special guestProgress & adoptionFuture work
![Page 5: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/5.jpg)
Intro
![Page 6: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/6.jpg)
What is PubSubHubbub?
A simple, topic-based publish/subscribe protocol Turns Atom and RSS feeds into real-time streamsA single API for web-scale, low-latency messagingThree participants: Publisher, Subscriber, Hubs
Publisher SubscriberHub
![Page 7: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/7.jpg)
Design goals
Decentralized: No one company in controlScale to the size of the whole webPublishing and subscribing as easy as possiblePush any complexity towards the HubPragmatic (i.e., not theoretically perfect, but solve huge, known use cases with minimal effort)
![Page 8: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/8.jpg)
Why another protocol?
Almost every company already has an internal systemTIBCO, WebsphereMQ, ActiveMQ, RabbitMQ, ...Proprietary message payloads, topics, networks
Existing attempts at a standard haven't caught on
XMPP started in 1999, still isn't used for interop widely beyond IM (may change with OneSocialWeb.org?)Overkill: XEP-0060, WS-*, AMQP, RestMS, new REST-*
![Page 9: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/9.jpg)
How-to for publishers
1. Add a declaration in your feed with Hub(s) of choice<link rel="hub" href="https://pubsubhubbub.appspot.com/"/>
2. Add something to your feed!
3. Send a ping to the Hub(s) with the feed URLPOST / HTTP/1.1Content-Type: application/x-www-form-urlencoded...
hub.mode=publish&hub.url=<your feed>
4. 204 = Success, 4xx = Bad request, 5xx = Try again
![Page 10: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/10.jpg)
Publisher best practices
Use URLs for server-side filteringhttp://example.com/stuff?zip=94105
Use URLs for authorization
![Page 11: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/11.jpg)
How-to for subscribers
1. Detect the Hub(s') declaration in a feed 2. Send a subscribe request to the feed's Hub(s)
POST / HTTP/1.1Content-Type: application/x-www-form-urlencoded ...hub.mode=subscribe&hub.verify=sync&hub.topic=<feed URL>&hub.callback=<callback URL>
3. Hub(s) will send a request to verify the subscriptionGET /callback?hub.challenge=<random> HTTP/1.1
HTTP/1.1 200... <echo random>
![Page 12: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/12.jpg)
How-to for subscribersReceive new content from the Hub(s)
POST /callback HTTP/1.1Content-Type: application/atom+xml...<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom"> <title>Awesome feed</title> <link rel="hub" href="http://pubsubhubbub.appspot.com"/> ... <entry> ... </entry></feed>
![Page 13: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/13.jpg)
The role of a hub
FunctionsAccept and verify subscriptions to new topicsReceive pings from publishers, retrieve contentExtract new/updated items from feedSend all subscribers the new contentDoS protections
![Page 14: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/14.jpg)
The role of a hub
Logical componentPublishers may be their own HubCombined Hub/Publisher has p2p speed-up
Quality
ScalabilityReliability
![Page 15: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/15.jpg)
Julien Genestoux, Superfeedr
![Page 16: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/16.jpg)
PubSubHubbub at Superfeedr
Google IO, May 2010
![Page 17: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/17.jpg)
Superfeedr
1. Default hub
2. Hosted hubs
3. PubSubHubbub + Benefits
![Page 18: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/18.jpg)
Default Hub
Historical Superfeedr
Avoid pollingSmart scheduling
Protocol mapping : RSSCloud, SUP, XML-RPC ping...
Push to subscribers (XMPP too :D)
Schema mapping
![Page 19: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/19.jpg)
Default Hub
Focus on what really makes a difference : your core business!
![Page 20: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/20.jpg)
Use-casesiPhone Notification : Urban Airship, Boxcar
Feed reader : Webwag, Feedingo
Desktop Notification : Adobe Wave
Semantic search : Guzzle.it, Twingly!
Social Web : SixApart
![Page 21: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/21.jpg)
Hosted Hubs
Don’t re-invent the wheel
Don’t run/maintain/debug the wheel
Your hub, YOUR data
Analytics, callbacks and more
![Page 22: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/22.jpg)
References
Blogging
Social Nets
Media
![Page 23: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/23.jpg)
Schema MappingTons of different formats : RSS X, Atom Y
Tons of different namespaces : Digg vs. Mixx vs. Yahoo Buzz. Same semantics
Tons of invalid stuff (missing tags, date, unique id..)
Location : Geo-RSS
Social
![Page 24: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/24.jpg)
Extensions
Digest Notifications (Heartbeat + Digest)
Feed status (querying superfeedr)
Subscription callback
Virtual feeds
![Page 25: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/25.jpg)
Infrastructure
Botnet!
Independent XMPP workers with their own lifecycle.
Massive “Ring” for scheduling
Clustered cache for diff-ing
![Page 26: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/26.jpg)
A few numbers
Content pushed to 1.8M endpoints
20M+ of Atom entries pushed daily
~50 hosted hubs
45 “dispatchers”
80 “parsers”
~50 servers
![Page 27: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/27.jpg)
A few numbers
![Page 28: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/28.jpg)
TODO
Make your feeds realtime
Follow @superfeedr
Say hello to @julien51Pick Stickers
Thanks
![Page 29: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/29.jpg)
Progress
![Page 30: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/30.jpg)
Adoption
Over 100 Million feeds are PubSubHubbub-enabled
Companies: Superfeedr (and friends), Google, Six Apart, LiveJournal, MySpace, TwitterFeed, Netvibes, Cliqset, Gnip, PostRank, ...
Google products: Buzz, FeedBurner, Blogger, Reader shared items, Google Alerts, Fastflip, ...
![Page 31: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/31.jpg)
Fun numbers from the reference Hub
200+ feed fetches per second (peak avg.)250+ items delivered per second (peak avg.)
Includes item updates 70 million active subscriptions1.2 billion items seen since July 2009
![Page 32: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/32.jpg)
http://pubsubhubbub.googlecode.com
Publisher clients: Perl, PHP, Python, Ruby, Java, Haskell, C#, MovableType, WordPress, Melody, Django, Zend, Drupal
Subscriber clients/frameworks: PHP, .NET, Scala, Zend, Drupal, Django, Tornado, App Engine, NodeJS, Rails
Hubs: App Engine, WordPress, Erlang, Twisted Python, Ruby, Perl, Django
Active mailing list with 440+ members More publishers, subscribers, hubs on the way
![Page 33: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/33.jpg)
Future work
![Page 34: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/34.jpg)
In progress
Arbitrary content types (JSON, HTML, XML)Microformats folks want HTML pushGoogle wants XML Sitemaps updatesPlan to build on LRDD web linkingFacebook uses 1/2 of PuSH for their new APIs
![Page 35: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/35.jpg)
In progress
Private feedsFully encrypted, authorized, authenticatedIntegration with OAuth, WebFingerApply business policies Per-item privacy control
![Page 36: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/36.jpg)
Further reading
OStatushttp://ostatus.org/
Buzz APIhttp://code.google.com/apis/buzz/
Facebook real-time APIhttp://developers.facebook.com/docs/api/realtime
![Page 37: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/37.jpg)
View live notes and ask questions about this session on Google Wave
http://tinyurl.com/push-io2010Mehttp://onebigfluke.com
Superfeedrhttp://superfeedr.com
![Page 38: Social Real Time Apps Pub Sub Hubbub](https://reader031.fdocuments.us/reader031/viewer/2022020122/546b3c73b4af9f2d708b45cc/html5/thumbnails/38.jpg)