riak@ideeli

Aaron Brown, Lead Systems Engineerabrown@ideeli.com@aaronbbrown777

Thursday, February 14, 13

What is an ideeli?

Thursday, February 14, 13

Flash Saleshttp://www.flickr.com/photos/moonlightbulb/4064060773/

Thursday, February 14, 13

Flash SalesThursday, February 14, 13

Flash SalesThursday, February 14, 13

Flash SalesThursday, February 14, 13

Flash SalesThursday, February 14, 13

Obligatory Stats

• 2,000,000 visitors/week

• 80,000 front end requests/minute

• 120 production servers

• 300 deploys a year

• 40ms front end response time

Thursday, February 14, 13

Obligatory Riak Stats

• 1.2MM keys

• 4 virtualized nodes

• 4ms response

• 40,000 GETs/min

• 2,000 PUTs/min

Thursday, February 14, 13

The SpikeThursday, February 14, 13

The StackThursday, February 14, 13

New Toys Are Fun(9AM - 5PM)

Thursday, February 14, 13

What Counts1AM - 7AM

Thursday, February 14, 13

K.I.S.S.Keep It Simple Stupid

Thursday, February 14, 13

K.I.S.S.Keep It Simple Stupid

Kan It Sell Shoes?

Thursday, February 14, 13

3 Uses for Riak

• Cache Warming

Thursday, February 14, 13

3 Uses for Riak

• Cache Warming

• Feature Testing (LABS)

Thursday, February 14, 13

3 Uses for Riak

• Cache Warming

• Feature Testing (LABS)

• User Favorites

Thursday, February 14, 13

Cache Warming

Thursday, February 14, 13

The SpikeThursday, February 14, 13

Cache WarmingPseudo-code:

every.30.minutes do  all_urls.each do |url|    curl "#{url}?force_cache=true"  endend

Thursday, February 14, 13

Cache WarmingThursday, February 14, 13

Response timesThursday, February 14, 13

History

Thursday, February 14, 13

memcached

Thursday, February 14, 13

memcached~40ms

Thursday, February 14, 13

memcached5/6 requests ~ 40ms

1/6 requests ~ 3000ms=

~500ms average

Thursday, February 14, 13

Domino Effecthttp://www.flickr.com/photos/soham_pablo/309379628/

Thursday, February 14, 13

Manual InterventionThursday, February 14, 13

membase(couchbase)

Thursday, February 14, 13

RiakAll requests ~40ms

Thursday, February 14, 13

RiakAll requests ~40ms

Thursday, February 14, 13

LABS

• Feature Testing Framework

• Key based on user id

• Value YAML string w/ list of experiments

Thursday, February 14, 13

FavoritesThursday, February 14, 13

FavoritesThursday, February 14, 13

Favorites

• Partially normalized

• user_#{id} → array of productsproduct_#{id} → metadata about product

Thursday, February 14, 13

Rails Integration

• Modified riak-client gem for Rails 2.3

• Load balancing built in

• Multiple primary and secondary servers

• Custom ActiveSupport::Cache store

Thursday, February 14, 13

Rails Integrationmodule ActiveSupport  module Cache    class RailsStore < Store

      def read      end

      def write      end

      def delete      end  endend

config.after_initialize do  ActionController::Base.cache_store = Rails.rails_store.fragmentsend

Thursday, February 14, 13

MonitoringThursday, February 14, 13

GraphiteThursday, February 14, 13

New RelicThursday, February 14, 13

Alerting

• Compute random string

• PUT value into Riak

• GET value out of Riak

• alert if value_in != value_out

• alert if operation took too long

Thursday, February 14, 13

Tuning

Thursday, February 14, 13

Bitcask Merges

Thursday, February 14, 13

Disk UsageThursday, February 14, 13

Heavy I/OThursday, February 14, 13

expiry_grace_timeThursday, February 14, 13

Offset Merge WindowsThursday, February 14, 13

Puppetclass ideeli_riak ( $merge_window_start, $merge_window_end ) {... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), }...}

class riak-node { $merge_window_start = inline_template( ‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>' )

class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 }}

Thursday, February 14, 13

Riak Loves Disk

Thursday, February 14, 13

New Disk ArrayThursday, February 14, 13

New Disk ArrayThursday, February 14, 13

Improvement!Thursday, February 14, 13

We’re Hiringhttp://ideelicareers.com/

Thursday, February 14, 13

Questions?

Thursday, February 14, 13