Twitter: @ketab n egtema3.pdf · Twitter: @ketab_n. Twitter: @ketab_n. Twitter: @ketab_n
Scaling Twitter 12758
-
Upload
davidblum -
Category
Technology
-
view
3.687 -
download
0
description
Transcript of Scaling Twitter 12758
![Page 1: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/1.jpg)
Big Bird.(scaling twitter)
![Page 2: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/2.jpg)
Rails Scales.(but not out of the box)
![Page 3: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/3.jpg)
First, Some Facts
• 600 requests per second. Growing fast.
• 180 Rails Instances (Mongrel). Growing fast.
• 1 Database Server (MySQL) + 1 Slave.
• 30-odd Processes for Misc. Jobs
• 8 Sun X4100s
• Many users, many updates.
![Page 4: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/4.jpg)
![Page 5: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/5.jpg)
![Page 6: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/6.jpg)
![Page 7: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/7.jpg)
Oct Nov Dec Jan Feb March Apr
Joy Pain
![Page 8: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/8.jpg)
IM IN UR RAILZ
MAKIN EM GO FAST
![Page 9: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/9.jpg)
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse
It’s Easy, Really.
![Page 10: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/10.jpg)
1. Realize Your Site is Slow
2. Optimize the Database
3. Cache the Hell out of Everything
4. Scale Messaging
5. Deal With Abuse
6. Profit
It’s Easy, Really.
![Page 11: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/11.jpg)
{ Part the First }
themoreyouknow
![Page 12: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/12.jpg)
We Failed at This.
![Page 13: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/13.jpg)
Don’t Be Like Us
• Munin
• Nagios
• AWStats & Google Analytics
• Exception Notifier / Exception Logger
• Immediately add reporting to track problems.
![Page 14: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/14.jpg)
Test Everything
• Start Before You Start
• No Need To Be Fancy
• Tests Will Save Your Life
• Agile Becomes Important When Your Site Is Down
![Page 15: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/15.jpg)
Benchmarks?let your users do it.
<!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:00 in 409 ms (d 88 / r 307). thank you, come again. -->
<!-- served to you through a copper wire by raven.twitter.com at 22 Apr 15:01 in 450 ms (d 96 / r 337). thank you, come again. -->
<!-- served to you through a copper wire by quetzal at 22 Apr 15:01 in 384 ms (d 70 / r 297). thank you, come again. -->
<!-- served to you through a copper wire by sampaati at 22 Apr 15:02 in 343 ms (d 102 / r 217). thank you, come again. -->
<!-- served to you through a copper wire by kolea.twitter.com at 22 Apr 15:02 in 235 ms (d 87 / r 130). thank you, come again. -->
<!-- served to you through a copper wire by firebird at 22 Apr 15:03 in 2094 ms (d 643 / r 1445). thank you, come again. -->
![Page 16: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/16.jpg)
The Database{ Part the Second }
![Page 17: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/17.jpg)
“The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
![Page 18: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/18.jpg)
“The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
![Page 19: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/19.jpg)
“The Next Application I Build is Going to Be Easily Partitionable” - S. Butterfield
![Page 20: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/20.jpg)
Too Late.
![Page 21: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/21.jpg)
Index Everything
![Page 22: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/22.jpg)
class AddIndex < ActiveRecord::Migration def self.up add_index :users, :email end
def self.down remove_index :users, :email endend
Repeat for any column that appears in a WHERE clause
Rails won’t do this for you.
![Page 23: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/23.jpg)
Denormalize A Lot
![Page 24: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/24.jpg)
class DenormalizeFriendsIds < ActiveRecord::Migration def self.up add_column "users", "friends_ids", :text end
def self.down remove_column "users", "friends_ids" endend
![Page 25: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/25.jpg)
class Friendship < ActiveRecord::Base belongs_to :user belongs_to :friend
after_create :add_to_denormalized_friends after_destroy :remove_from_denormalized_friends
def add_to_denormalized_friends user.friends_ids << friend.id user.friends_ids.uniq! user.save_without_validation end
def remove_from_denormalized_friends user.friends_ids.delete(friend.id) user.save_without_validation endend
![Page 26: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/26.jpg)
Don’t be Stupid
![Page 27: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/27.jpg)
bob.friends.map(&:email)Status.count()
“email like ‘%#{search}%’”
![Page 28: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/28.jpg)
That’s where we are.
Seriously.If your Rails application is doing anything more
complex than that, you’re doing something wrong*.
* or you observed the First Rule of Butterfield.
![Page 29: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/29.jpg)
Partitioning Comes Later.(we’ll let you know how it goes)
![Page 30: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/30.jpg)
The Cache{ Part the Third }
![Page 31: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/31.jpg)
MemCache
![Page 32: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/32.jpg)
MemCache
![Page 33: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/33.jpg)
MemCache
![Page 34: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/34.jpg)
!
![Page 35: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/35.jpg)
class Status < ActiveRecord::Base class << self def count_with_memcache(*args) return count_without_memcache unless args.empty? count = CACHE.get(“status_count”) if count.nil? count = count_without_memcache CACHE.set(“status_count”, count) end count end alias_method_chain :count, :memcache end after_create :increment_memcache_count after_destroy :decrement_memcache_count ...end
![Page 36: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/36.jpg)
class User < ActiveRecord::Base def friends_statuses ids = CACHE.get(“friends_statuses:#{id}”) Status.find(:all, :conditions => [“id IN (?)”, ids]) endend
class Status < ActiveRecord::Base after_create :update_caches def update_caches user.friends_ids.each do |friend_id| ids = CACHE.get(“friends_statuses:#{friend_id}”) ids.pop ids.unshift(id) CACHE.set(“friends_statuses:#{friend_id}”, ids) end endend
![Page 37: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/37.jpg)
Active
Recor
d
The Future
![Page 38: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/38.jpg)
90% API RequestsCache Them!
![Page 39: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/39.jpg)
“There are only two hard things in CS: cache invalidation and naming things.”
– Phil Karlton, via Tim Bray
![Page 40: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/40.jpg)
Messaging{ Part the Fourth }
![Page 41: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/41.jpg)
You Already Knew All That Other Stuff, Right?
![Page 42: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/42.jpg)
ProducerProducerProducer
MessageQueue
ConsumerConsumerConsumer
![Page 43: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/43.jpg)
DRb
• The Good:
• Stupid Easy
• Reasonably Fast
• The Bad:
• Kinda Flaky
• Zero Redundancy
• Tightly Coupled
![Page 44: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/44.jpg)
Jabber Client(drb)
PresenceIncomingMessages
OutgoingMessages
ejabberd
MySQL
![Page 45: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/45.jpg)
ServerDRb.start_service ‘druby://localhost:10000’, myobject
Clientmyobject = DRbObject.new_with_uri(‘druby://localhost:10000’)
![Page 46: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/46.jpg)
Rinda
• Shared Queue (TupleSpace)
• Built with DRb
• RingyDingy makes it stupid easy
• See Eric Hodel’s documentation
• O(N) for take(). Sigh.
![Page 47: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/47.jpg)
SELECT * FROM messages WHERE substring(truncate(id,0),-2,1) = #{@fugly_dist_idx}
Timestamp: 12/22/06 01:53:14 (4 months ago)Author: latticeMessage: Fugly. Seriously. Fugly.
![Page 48: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/48.jpg)
It Scales.(except it stopped on Tuesday)
![Page 49: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/49.jpg)
Options
• ActiveMQ (Java)
• RabbitMQ (erlang)
• MySQL + Lightweight Locking
• Something Else?
![Page 50: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/50.jpg)
erlang?
What are you doing?
Stabbing my eyes out with a fork.
![Page 51: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/51.jpg)
Starling
• Ruby, will be ported to something faster
• 4000 transactional msgs/s
• First pass written in 4 hours
• Speaks MemCache (set, get)
![Page 52: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/52.jpg)
Use Messages to Invalidate Cache
(it’s really not that hard)
![Page 53: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/53.jpg)
Abuse{ Part the Fifth }
![Page 54: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/54.jpg)
The Italians
![Page 55: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/55.jpg)
9000 friends in 24 hours(doesn’t scale)
![Page 56: Scaling Twitter 12758](https://reader034.fdocuments.us/reader034/viewer/2022051608/5456f028af7959755d8b4e22/html5/thumbnails/56.jpg)
http://flickr.com/photos/heather/464504545/http://flickr.com/photos/curiouskiwi/165229284/http://flickr.com/photo_zoom.gne?id=42914103&size=lhttp://flickr.com/photos/madstillz/354596905/http://flickr.com/photos/laughingsquid/382242677/http://flickr.com/photos/bng/46678227/