Advanced Performance Optimization of Rails …assets.en.oreilly.com/1/event/24/Advanced...

Advanced Performance Optimization

of Rails Applications

Alexander DymoRailsConf 2009

www.acunote.com

http://en.oreilly.com/rails2009/public/schedule/detail/8615

Project Management and Scrum Software

2 / 84Alexander Dymo • Advanced Performance Optimization of Rails Applications • RailsConf 2009

Project Management and Scrum SoftwareWhat Am I Optimizing?

Acunote www.acunote.com

Online project management and Scrum software

Ruby on Rails application since inception in 2006

● ~3700 customers

● Hosted on Engine Yard

● Hosted on Customers' Servers

● nginx + mongrel

● PostgreSQL

http://en.oreilly.com/rails2009/public/schedule/detail/8615


Project Management and Scrum SoftwarePerformance Degradation Over Time

100

150

200

250

300

April 2008 May 2008 June 2008 July 2008

Req

uest

Tim

e (o

n de

velo

pmen

t bo

x),

%

Actually Happens: O(nc)

Best Case: O(log n)


Project Management and Scrum SoftwareSolutions?

Throw Some Hardware at it!


Project Management and Scrum SoftwareSolutions?

Performance Optimization!



What to optimize?


Project Management and Scrum SoftwareWhat To Optimize?

Development?


Project Management and Scrum SoftwareWhat To Optimize?

Development

AND Production



How to optimize?


Project Management and Scrum SoftwareHow To Optimize?

Three rules of

performance optimization


Project Management and Scrum SoftwareThree Rules Of Performance Optimization

1. Measure!...the universal experience of programmers who have been using measurement tools has been that their intuitive guesses fail...Knuth



2. Optimize only what's slow!



3. Optimize for the user!


Project Management and Scrum SoftwareThings To Optimize

● Development– Ruby code

– Rails code

– Database queries

– Alternative Ruby

● Production– Shared filesystems and databases

– Live debugging

– Load balancing

● Frontend– HTTP

– Javascript

– Internet Explorer


Project Management and Scrum SoftwareOptimizing Ruby: Date Class

What's wrong with Date?

> puts Benchmark.realtime { 1000.times { Time.mktime(2009, 5, 6, 0, 0, 0) } }

0.005

> puts Benchmark.realtime { 1000.times { Date.civil(2009, 5, 6) } }

0.080

16x slower than Time! Why?

%self total self wait child calls name

7.23 0.66 0.18 0.00 0.48 18601 <Class::Rational>#reduce

6.83 0.27 0.17 0.00 0.10 5782 <Class::Date>#jd_to_civil

6.43 0.21 0.16 0.00 0.05 31528 Rational#initialize

5.62 0.23 0.14 0.00 0.09 18601 Integer#gcd



Fixing Date: Use C, Luke!

Date::Performance gem with Date partially rewritten in C

by Ryan Tomayko (with my patches in 0.4.7)

> puts Benchmark.realtime { 1000.times { Time.mktime(2009, 5, 6, 0, 0, 0) } }

0.005

> puts Benchmark.realtime { 1000.times { Date.civil(2009, 5, 6) } }

0.080

> require 'date/performance'

puts Benchmark.realtime { 1000.times { Date.civil(2009, 5, 6) } }

0.006

git clone git://github.com/rtomayko/date-performance.gitrake package:build

cd dist && gem install date-performance-0.4.7.gem



Real-world impact of Date::Performance:

Before: 0.95 sec

After: 0.65 sec

1.5x!


Project Management and Scrum SoftwareOptimizing Ruby: Misc

Use String::<< instead of String::+=

> long_string = "foo" * 100000

> Benchmark.realtime { long_string += "foo" }

0.0003

> Benchmark.realtime { long_string << "foo" }

0.000004

Avoid BigDecimal comparisons with strings and integers

> n = BigDecimal("4.5")

> Benchmark.realtime { 10000.times { n <=> 4.5 } }

0.063

> Benchmark.realtime { 10000.times { n <=> BigDecimal("4.5") } }

0.014

in theory: 4.5xin practice: 1.15x

in theory: 75xin practice: up to 70x




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimizing Rails: Preloading Associations

Preloading associations is always a good idea...

except when Rails can't do the job:

class Foobelongs_to :bar

end

foos = Foo.find_by_sql('select * from foos inner join bar')foos.first.bar #extra SQL query!


Project Management and Scrum SoftwareOptimizing Rails: Preloading Associations

Virtual Attributes plugin:

http://github.com/acunote/virtual_attributes/

class Barendclass Foo

belongs_to :barpreloadable_association :bar

end

foos = Foo.find_by_sql('select * from foos

left outer join (select id as preloaded_bar_id,

name as preloaded_bar_name from bars) as bars

on foos.bar_id = bars.preloaded_bar_id')foos.first.bar #no extra SQL query!


Project Management and Scrum SoftwareOptimizing Rails: String Callbacks

What can be wrong with this code?

class Task < ActiveRecord::Base before_save "some_check()"end...100.times { Task.create attributes}

Kernel#binding is called to eval() the string callback

That will duplicate your execution context in memory!

More memory taken => More time for GC


Project Management and Scrum SoftwareOptimizing Rails: Partial Rendering

Not too uncommon, right?

<% for object in objects %> #1000 times<%= render :partial => 'object',

:locals => { :object => object } %><% end %>

We create 1000 View instances for each object here!

Why?



list.rhtml

Optimizing Rails: Partial Rendering

Template inlining for the resque:

<% for object in objects %> #1000 times<%= render :partial => 'object',

:locals => { :object => object },:inline => true %>

<% end %>

list.rhtml

_object.rhtml

_object.rhtml

_object.rhtml

_object.rhtml

_object.rhtml

_object.rhtml

_object.rhtml

_object.rhtml


Project Management and Scrum SoftwareOptimizing Rails: Partial Rendering

Template Inliner plugin:

http://github.com/acunote/template_inliner/

Real world effect from template inlining:

Rendering of 300 objects, 5 partials for each object

without inlining: 0.89 sec

with inlining: 0.75 sec

1.2x




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimizing Database

How to optimize PostgreSQL:

explain analyze

explain analyze

explain analyze

...


Project Management and Scrum SoftwareOptimizing Database: PostgreSQL Tips

EXPLAIN ANALYZE explains everything, but...

... run it also for the "cold" database state!

Example: complex query which works on 230 000 rows and

does 9 subselects / joins:

cold state: 28 sec, hot state: 2.42 sec

Database server restart doesn't help

Need to clear disk cache:

sudo echo 3 | sudo tee /proc/sys/vm/drop_caches

(Linux)


Project Management and Scrum SoftwareOptimizing Database: PostgreSQL Tips

Use any(array ()) instead of in()

to force subselect and avoid join

explain analyze select * from issues where id in (select issue_id from tags_issues);

QUERY PLAN------------------------------------------------------------------------------------------------------------------------------------------------------- Merge IN Join (actual time=0.096..576.704 rows=55363 loops=1) Merge Cond: (issues.id = tags_issues.issue_id) -> Index Scan using issues_pkey on issues (actual time=0.027..270.557 rows=229991 loops=1) -> Index Scan using tags_issues_issue_id_key on tags_issues (actual time=0.051..73.903 rows=70052loops=1) Total runtime: 605.274 ms

explain analyze select * from issues where id = any( array( (select issue_id from tags_issues) ) );

QUERY PLAN------------------------------------------------------------------------------------------------------------------------------ Bitmap Heap Scan on issues (actual time=247.358..297.932 rows=55363 loops=1) Recheck Cond: (id = ANY ($0)) InitPlan -> Seq Scan on tags_issues (actual time=0.017..51.291 rows=70052 loops=1) -> Bitmap Index Scan on issues_pkey (actual time=246.589..246.589 rows=70052 loops=1) Index Cond: (id = ANY ($0)) Total runtime: 325.205 ms

2x!


Project Management and Scrum SoftwareDatabase Optimization: PostgreSQL Tips

Push down conditions into subselects and joins

PostgreSQL often won't do that for you

select *,(select notes.author from notes where notes.issue_id = issues.id) as note_authors

from issueswhere org_id = 1

select *,(select notes.author from notes

where notes.issue_id = issues.id and org_id = 1) as note_authors

from issueswhere org_id = 1

Issuesid serialname varcharorg_id integer

Notesid serialname varcharissue_id integerorg_id integer


Project Management and Scrum SoftwareWhat To Do?

● Optimize For Development Box– Ruby code

– Rails code



● Optimize For Production– Shared filesystems and databases

– Live debugging

– Load balancing

● Optimize For The User– HTTP

– Javascript



Project Management and Scrum SoftwareAlternative Ruby

Everybody says

"JRuby and Ruby 1.9 are faster"

Is that true in production?



In short, YES!

= Acunote Benchmarks = MRI JRuby 1.9.1

Date/Time Intensive Ops 1.23 0.58 0.53

Rendering Intensive Ops 0.61 0.44 0.30

Calculations Intensive Ops 2.57 1.79 1.33

Database Intensive Ops 5.58 4.63 3.29



In short, YES!

= Acunote Benchmarks = MRI JRuby 1.9.1

Date/Time Intensive Ops 1x 2.1x 2.3x

Rendering Intensive Ops 1x 1.4x 2.0x

Calculations Intensive Ops 1x 1.4x 1.9x

Database Intensive Ops 1x 1.2x 1.7x

JRuby: 1.5x faster

Ruby 1.9: 2.0x faster



What is faster?



This is where all the improvement went:

Acunote Copy Tasks Benchmark MRI JRuby 1.9.1

Request Time 5.52 4.45 3.24

Template Rendering Time 0.53 0.21 0.21

Database Time 0.70 1.32 0.69

GC Time 1.07 N/A 0.62

Much faster template rendering!

Less GC!

JDBC database driver performance issue with JRuby?



Why faster?



Things I usually see in the profiler after optimizing: %self self calls name 2.73 0.05 351 Range#each-1 2.73 0.05 33822 Hash#[]= 2.19 0.04 4 Acts::AdvancedTree::Tree#walk_tree 2.19 0.04 44076 Hash#[] 1.64 0.03 1966 Array#each-1 1.64 0.03 378 Org#pricing_plan 1.64 0.03 1743 Array#each 1.09 0.02 1688 ActiveRecord::AttributeMethods#respond_to? 1.09 0.02 1311 Hash#each 1.09 0.02 6180 ActiveRecord::AttributeMethods#read_attribute_before_typecast 1.09 0.02 13725 Fixnum#== 1.09 0.02 46736 Array#[] 1.09 0.02 15631 String#to_s 1.09 0.02 24330 String#concat 1.09 0.02 916 ActiveRecord::Associations#association_instance_get 1.09 0.02 242 ActionView::Helpers::NumberHelper#number_with_precision 1.09 0.02 7417 Fixnum#to_s



# of method calls during one request:

50 000 - Array

35 000 - Hash

25 000 - String

Slow classes written in Ruby:

Date

Rational



Alternative Rubys optimize mostly:

● the cost of function call

● complex computations in pure Ruby

● memory by not keeping source code AST



So, shall I use alternative Ruby?

Definitely Yes!... but

JRuby: if your application works with it

(run requests hundreds of times to check)

Ruby 1.9: if all gems you need are ported




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimizing For Shared Environment

Issues we experienced deploying on Engine Yard:

1) VPS is just too damn slow

2) VPS may have too little memory to run the request!

3) shared database server is a problem

4) network filesystem may cause harm as well



VPS may have too little memory to run the request

Think 512M should be enough?

Think again.

We saw requests that took 1G of memory!

Solutions:

● buy more memory

● optimize memory

● set memory limits for mongrels (with monit)



You're competing for memory cache on a shared server:

1. two databases with equal load share the cache



You're competing for memory cache on a shared server:

2. one of the databases gets more load and wins the cache



As a result, your database can always be in a "cold" state

and you read data from disk, not from memory!

complex query which works on 230 000 rows and

does 9 subselects / joins:

from disk: 28 sec, from memory: 2.42 sec

Solutions:

optimize for the cold state

push down SQL conditions

sudo echo 3 | sudo tee /proc/sys/vm/drop_caches



fstat() is slow on network filesystem (GFS)

Request to render list of tasks in Acunote:

on development box: 0.50 sec

on production box: 0.50 - 2.50 sec




Couldn't figure out why until we ran strace

We used

a) filesystem store for fragment caching

b) expire_fragment(regexp)

Later looked through all cache directories even though we knew

the cache is located in only one specific subdir




Solution:

memcached instead of filesystem

if filesystem is ok, here's a trick:

http://blog.pluron.com/2008/07/hell-is-paved-w.html




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareLive Debugging

To see what's wrong on "live" application:

For Linux: strace and oprofile

For Mac and Solaris: dtrace

For Windows: uhm... about time to switch ;)

To monitor for known problems:

monit

nagios

own scripts to analyze application logs




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareLoad Balancing

The problem of round-robin and fair load balancing

Rails App 1

Rails App 2

Rails App 3

1

1 3

21 3

per-process queues

2

2



Solution: the global queue

Rails App 1

Rails App 2

Rails App 3

21 4 53



Think you need mod_rails / Passenger for this?

You're right, but...

you can emulate this with nginx and mongrels



Dedicated queues for long-running requests

Rails App 1

Rails App 2

Rails App 3

1

1

21 3

queue for long-running requests

2

regular per-process queues



nginx configuration for dedicated queues

upstream mongrel { server 127.0.0.1:5000; server 127.0.0.1:5001;}upstream rss_mongrel {

server 127.0.0.1:5002;}server { location / { location ~ ^/feeds/(rss|atom) { if (!-f $request_filename) { proxy_pass http://rss_mongrel; break; } } if (!-f $request_filename) { proxy_pass http://mongrel;

} }}




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimize For The User: HTTP

Network and FrontendBackend

Things to consider:

● Gzip HTML, CSS and JS

● Minify JS

● Collect JS and CSS

(javascript_include_tag :all, :cache => true)

● Far future expires headers for JS, CSS, images

● Sprites

● Cache-Control: public

● everything else YSlow tells you

5% 95%




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimize Frontend: Javascript

The worst enemy of every

web developer is...



Things you don't want to hear from your users:

"...Your server is slow..."

said the user after clicking

on the link to show a form

with plain javascript (no AJAX)



Known hotspots in Javascript:

- eval()

- all DOM operations - avoid if possible, for example

- use element.className instead of element.readAttribute('class')

- use element.id instead of element.readAttirbute('id')

- $$() selectors, especially attribute selectors

- may be expensive, measure first

- $$('#some .listing td a.popup[accesslink]'

- use getElementsByTagName() and iterate results instead

- element.style.* changes

- change class instead

- $() and getElementById on large (~20000 elements) pages




– Rails code




– Live debugging

– Load balancing


– Javascript



Project Management and Scrum SoftwareOptimize Frontend: IE

Unsurprisingly...

IE is also slow!

Slow things that are especially slow in IE:

- $() and $$(), even on small pages

- getElementsByName()

- style switching


Project Management and Scrum SoftwareOptimize Frontend: IE

Good things about IE:

profiler in IE8

fast in IE => fast everywhere else!


Project Management and Scrum SoftwareKeep It Fast!

So, you've optimized your application.

How to keep it fast?


Project Management and Scrum SoftwareKeep It Fast!

Measure, measure and measure...

Use profiler

Optimize CPU and Memory


Project Management and Scrum SoftwareKeep It Fast: Measure

Keep a set of benchmarks for most frequent user requests.For example:

Benchmark Burndown 120 0.70 ± 0.00Benchmark Inc. Burndown 120 0.92 ± 0.01Benchmark Sprint 20 x (1+5) (C) 0.45 ± 0.00Benchmark Issues 100 (C) 0.34 ± 0.00Benchmark Prediction 120 0.56 ± 0.00Benchmark Progress 120 0.23 ± 0.00Benchmark Sprint 20 x (1+5) 0.93 ± 0.00Benchmark Timeline 5x100 0.11 ± 0.00Benchmark Signup 0.77 ± 0.00Benchmark Export 0.20 ± 0.00Benchmark Move Here 20/120 0.89 ± 0.00Benchmark Order By User 0.98 ± 0.00Benchmark Set Field (EP) 0.21 ± 0.00Benchmark Task Create + Tag 0.23 ± 0.00

... 30 more ...



Benchmarks as a special kind of tests:

class RenderingTest < ActionController::IntegrationTest

def test_sprint_rendering login_with users(:user), "user"

benchmark :title => "Sprint 20 x (1+5) (C)", :route => "projects/1/sprints/3/show", :assert_template => "tasks/index" end

end

Benchmark Sprint 20 x (1+5) (C) 0.45 ± 0.00



Benchmarks as a special kind of tests:

def benchmark(options = {})(0..100).each do |i|

GC.startpid = fork do

beginout = File.open("values", "a")ActiveRecord::Base.transaction do

elapsed_time = Benchmark::realtime dorequest_method = options[:post] ? :post : :getsend(request_method, options[:route])

endout.puts elapsed_time if i > 0out.closeraise CustomTransactionError

endrescue CustomTransactionError

exitend

endProcess::waitpid pidActiveRecord::Base.connection.reconnect!

endvalues = File.read("values")print "#{mean(values).to_02f} ± #{sigma(values).to_02f}\n"

end


Project Management and Scrum SoftwareKeep It Fast: Query Testing

Losing 10ms in benchmark might seem OK

Except that it's sometimes not

because you're running one more SQL query



def test_queriesqueries = track_queries do

get :indexendassert_equal queries, [

"Foo Load","Bar Load","Event Create"

]end



module ActiveSupportclass BufferedLogger

attr_reader :tracked_queries

def tracking=(val) @tracked_queries = [] @tracking = val end

def debug_with_tracking(message) @tracked_queries << $1 if @tracking && message =~ /3[56]\;1m(.* (Load|Create|Update|Destroy)) \(/ debug_without_tracking(message) end alias_method_chain :debug, :tracking

endend

class ActiveSupport::TestCase def track_queries(&block) RAILS_DEFAULT_LOGGER.tracking = true yield result = RAILS_DEFAULT_LOGGER.tracked_queries RAILS_DEFAULT_LOGGER.tracking = false result endend


Project Management and Scrum SoftwareKeep It Fast: Use Profiler

Profiler will always tell you what's wrong:

%self total self child calls name 8.39 0.54 0.23 0.31 602 Array#each_index 7.30 0.41 0.20 0.21 1227 Integer#gcd 6.20 0.49 0.17 0.32 5760 Timecell#date 5.11 0.15 0.14 0.01 1 Magick::Image#to_blob

gem install ruby-prof

use "performance" tests for Rails 2.x

KCachegrind to visualize the results

http://kcachegrind.sourceforge.net


Project Management and Scrum SoftwareKeep It Fast: Use Profiler


Project Management and Scrum SoftwareKeep It Fast: Optimize CPU and Memory

Memory profiler will explain the missing details:

Example benchmark: 5.52 sec request time, 1.07 sec GC time

Consumed memory: 55M

Ruby runs GC after allocating

8M memory or doing 10000 allocations

Simple math: 55 / 8 = 6 GC calls

Each GC call takes 0.18 sec!


Project Management and Scrum SoftwareKeep It Fast: Optimize CPU and Memory

How to use memory profiler:

Recompile Ruby with GC patch

http://www.acunote.com/system/ruby186-p287-gc.patch

Reinstall ruby-prof

Use RUBY_PROF_MEASURE_MODE=memory

when running ruby-prof

http://blog.pluron.com/2008/02/memory-profilin.html


Project Management and Scrum SoftwareRemember!

Measure, measure, measure... (with ruby-prof)

Optimize only what's slow

Optimize not only CPU, but memory

Optimize for user experience

Keep a set of performance regression tests

Monitor performance


Project Management and Scrum SoftwareHelp Us Make Ruby and Rails Faster!

We need your help!

● we'd like to see GC patch applied to mainstream Ruby

(to profile memory without recompiling Ruby)

● we'd like to see people porting their gems to Ruby 1.9

● we'd like to see more real-world benchmarks



Thanks!

Slides for this talk: http://en.oreilly.com/rails2009/public/schedule/detail/8615

Rails performance articles and more:http://blog.pluron.com

Advanced Performance Optimization of Rails …assets.en.oreilly.com/1/event/24/Advanced...

Documents

Transcript of Advanced Performance Optimization of Rails …assets.en.oreilly.com/1/event/24/Advanced...