Rails on scale
-
Upload
yifat-kanfi -
Category
Documents
-
view
139 -
download
2
Transcript of Rails on scale
![Page 2: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/2.jpg)
Performance - what itactually is
well, code which does what it'ssupposed to, and doesn't do it asslow as rails 3.0's boot time.
in every part of a project's lifecycle, the way we treat performance is very different.
![Page 3: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/3.jpg)
When you're young
![Page 4: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/4.jpg)
When you're young and naive
when you start with a project, and it's still small on traffic, write naive code!
![Page 5: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/5.jpg)
Do TDD!
To avoid this :)
![Page 6: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/6.jpg)
Write short and concise code
![Page 7: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/7.jpg)
Don't bother with premature optimization
![Page 8: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/8.jpg)
(when you prematurely optimize, this happens)
![Page 9: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/9.jpg)
READ!
prepare for growth, because you're optimistic and all that. make sure you'll know what to do when shit gets real.
![Page 10: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/10.jpg)
be naive but not TOO naive, though
there are some things which just scream - don't do this! it's gonna suck, BAD!the n+1 query issue is a good example of too naive code.
![Page 11: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/11.jpg)
The controller
![Page 12: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/12.jpg)
The view
![Page 13: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/13.jpg)
The view
![Page 14: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/14.jpg)
The problemwe have an array of users, and when we iterate over that array we reach for profile_image and for posts, which triggers two queries to the DB for each user. ending up with 2n+1 queries, n being number of users
ActiveRecord's includes prefetches the extra queries, so they turn into two queries, instead of 2n queries
The solution
![Page 15: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/15.jpg)
The new controller
now there are only 3 queries, instead for 2n+1 (n being the amount of users)note that this might not be the right thing to do in larger scale projects. you might want to cache the profile image in redis, for instance, and completely avoid bringing in the profile_image object from the database.
![Page 16: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/16.jpg)
The importance of TDD
One of the roles I took upon arriving to FTBPro is kickstarting and leading the move to TDD, we also wrote a bunch of specs for our legacy code. Difference was incredible.
![Page 17: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/17.jpg)
Daily deploys
(instead of weekly deploys)
![Page 18: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/18.jpg)
New code's clean and awesome
![Page 19: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/19.jpg)
More focus on features
because code's fairly covered, there's less issues that come up in production (less being relative, yeah?)
![Page 20: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/20.jpg)
Upgrading made easy
we moved from rails 3.0 to 3.2 within two weeks. mostly because a vast majority of the issues were discovered in tests.
![Page 21: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/21.jpg)
But this talk is about performance!
When writing TDD, your code will be faster.● TDD forces you to write short and atomic
methods● we try to make these methods fast because
we hate slow specs :)● code doesn't fail on production, because if it
fails, we know about it before deployment.● no long-running methods, because they're
short and concise
![Page 22: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/22.jpg)
More performance specific TDD
using rspec you can test the time a method takes to run, set a threshold above which the spec fails!when using the bullet gem, you can set a limit on number of queries you allow a controller to run.Do benchmarks and performance tests
![Page 23: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/23.jpg)
original code - written without tests
![Page 24: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/24.jpg)
Rewrite - the specs
![Page 25: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/25.jpg)
![Page 26: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/26.jpg)
the actual code does exactly the same thing, but it's much shorter, and much more readable, because it's TDD, every method does only one thing, and is tested well.
![Page 27: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/27.jpg)
Conclusion - do TDD!
● code is shorter● easier to maintain● it's tested so when it breaks we know it
before it's on production● when we need to refactor or change it, we
can be fairly certain it will still work as intended because of the tests.
![Page 28: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/28.jpg)
When you're growing
![Page 29: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/29.jpg)
Now, you start growing, and there are growing pains
● because you've written TDD, when you optimize, you're not going to break anything (or are, but will see it when tests run)
● your code is short and concise, so optimizing it will be easy
● because you didn't optimize anything, you'll feel what needs to be optimized first (using newrelic and the such)
● again, don't optimize what's easy to optimize, optimize the parts which start causing pain.
![Page 30: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/30.jpg)
How to get the feelin`
![Page 31: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/31.jpg)
Newrelic
![Page 32: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/32.jpg)
shows you what's hurting the most
![Page 33: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/33.jpg)
And gives you a breakdown of that
![Page 34: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/34.jpg)
Google Analytics
![Page 35: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/35.jpg)
Browse your site (that crazy!!)
![Page 36: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/36.jpg)
Listen to users
they may come and complain, and may just go away. use google analytics to look for pages with unusually high bounce rate.
![Page 37: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/37.jpg)
Custom tools
statsd and graphite can be quite handy
![Page 38: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/38.jpg)
Real life example
in FTBPro, we have a score table for each league, it gets daily(ish) updated from an external source.We noticed in Newrelic that the league page took a long time to load. A short investigation pointed to the table, which led to a tiny change in the code.
![Page 39: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/39.jpg)
Before
![Page 40: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/40.jpg)
After
![Page 41: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/41.jpg)
What? wait! it looks the same!
well, almost. there are two changes - one is a tiny change in variable names to make code more readable.the second change is we used a caching mechanism to bring in the team (called Subject in our code) without making any queries.
the difference was HUGE. time to build the table when cache was cold went down from 7 seconds to 0.5 seconds.
![Page 42: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/42.jpg)
So - what have we done exactly?
● we removed an n+1 query not by including stuff, but by avoiding the query altogether
● we used a caching mechanism for teams, which takes the team's nick (Barcelona can be referred to as barca, or F.C. Barcelona) and returns the cached team.
● used that cache to speed up a very painful part of the site by a lot.
● and yes, of course the view is cached so the rebuild of the table only happens once a day.
![Page 43: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/43.jpg)
When you need to refactor, or rewrite.
refactoring is taking code and changing it, while rewriting is starting from scratch.different reasons for refactoring or rewriting● code is causing performance issues● code is too clumsy, and makes debugging
very hard and costly● code just looks horrid● Tom said so.But when do we rewrite and when is it enough just to refactor?
![Page 44: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/44.jpg)
When to refactor
● code is generally ok, maintainable and worth keeping
● small changes would get the desired result easily
● code is well covered with specs● we're too damn lazy to rewrite it all (yes, it's
a valid reason, lazy programmers create short code)
![Page 45: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/45.jpg)
When should we just throw it away and rewrite.
● if the maintaining the code costs more than rewriting it, rewrite, and do it well!
● if the code does not have any test coverage and is untestable.
● when code looks like the Flying Spaghetti Monster
● when it was written by Avi Tzurel :)
make sure that new code is good, if you rewrite shit code to new shit code, you've done nothing!
![Page 46: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/46.jpg)
A little bit about queues
DelayedJob, Resque, Sidekiq, they all got strange names with typos in them. They all save us from hell.
![Page 47: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/47.jpg)
Move long running stuff to the background!
Let's talk about user registration - a user comes to the site, signs in with facebook, we get his image, his facebook friends, etc. It takes a while, even a long while.
![Page 48: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/48.jpg)
Put it aside!
Calculating all that stuff is long.This doesn't have to be that way. We really only need to save the user's name, facebook details, and that's it. We'll do the rest in the background, using one of the queueing mechanisms Ruby has to offer us. This will allow us to give the user a better, faster experience.
![Page 49: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/49.jpg)
Starting to get seriously huge
(ok, maybe this isn't a good image)
![Page 50: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/50.jpg)
Hitting large scale
Q - when do you know you've hit large scale?A - when your servers crash daily.
now, when you've reached that, you know you need to do some really drastic stuff to adjust to your new position.
![Page 51: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/51.jpg)
A quick detour to the land of DevOps
● handling large scale requires a lot of resources, and managing these resources effectively.
● cloud services such as Amazon AWS give companies some simple tools to handle scale very well.
● but if you don't know what you're doing, call for help :)
![Page 52: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/52.jpg)
FTBpro's setup on AWS
![Page 53: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/53.jpg)
Mysql with RDS
RDS is Amazon's mysql. It's optimized and easy to set up. saves us a lot of time on system administration.
![Page 54: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/54.jpg)
Memcached with elasticache
Elasticache is the Amazon memcached service. same as RDS, saves us time bother of messing with memcached servers.
![Page 55: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/55.jpg)
Custom redis server
thinking about moving to cloud services to save us the trouble.
![Page 56: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/56.jpg)
Web servers with nginx+unicorn
nginx+unicorn are like milk and cookies. With the right setup we also get zero-downtime deploys, which are awesome.
![Page 57: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/57.jpg)
Resque servers
they're also built for automatic scaling. just because we're awesome!
![Page 58: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/58.jpg)
CDN cache with cotendo (akamai)
logged out users don't even touch the web servers - their content is served by the CDN.
![Page 59: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/59.jpg)
Build it for quick and automatic scale
● self-deploying servers - when you start the server from its image, it will deploy to itself and start serving traffic / run resque workers
● adding servers is automatic - when there's high traffic, start them up, then kill them when traffic's low
● this allows to pay the minimum for hosting, while keeping scalability
![Page 60: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/60.jpg)
careful with these self-deploying robots! make sure they know the robot rules...
The rules:1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.2. A robot must obey any orders given to it by human beings, except where such orders would conflict with the First Law.3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
![Page 61: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/61.jpg)
ok, back to Ruby (kind of)
When reaching massive scale, we'd start looking for custom solutions - relational dbs would stay forever, but some things should be moved to other customized solutions.● consider using mongo for document-like
data● consider using neo4j or other graph dbs for
representing graph data (sorry Avi, mongo ain't no graph DB!)
![Page 62: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/62.jpg)
And don't forget to stay naive!
being large scale, but still fun and lean, can be hard, but pulling it off is worth it!
![Page 63: Rails on scale](https://reader034.fdocuments.us/reader034/viewer/2022052412/558c1862d8b42ae2718b4635/html5/thumbnails/63.jpg)
Thanks for not falling asleep!