Getting started with Riak in the Cloud

I work with Code!

I work with Data!

Michael Brodhead Ines Sombra@RandomMood@mojinations

Agenda

Application layer

Database layer

A tale of 2 apps

•One brand new (no code written)

•One in production running MySQL

MultipleRegions

MultipleClouds

11

The Librariesriak-client

• simple Ruby access to Riak's HTTP & protocol buffer APIs

The Libraries

Ripple

• Higher level, ORM interface

• ActiveModel based buffer APIs

Simplicity

The Librariesdm-riak

• Already using DataMapper

• Hoping for trivial switchover

The LibrariesRiak Shim

• database.yml

• Bucket naming

• Hides JSON

• 2i

Challenges

•Unit tests

•Data migration

•Deployment

Testing

Testing

Deployment

Use your app logic

How do we get here?

... but we have no hardware

“We need a Riak cluster”

(@mojinations circa 3 months ago)

Let’s do it!

Why Riak Product?

• Linearly Scalable

• Fault-tolerant

•Ops friendly

• Product lacks HA options

Riak @ Engine Yard• We’ll use it first

• No custom Chef

• Provisioning, configuration, monitoring, & backups

Your Choices

• Number of nodes

• Instance type

• Data location

• Backend type

You are trying to create a Riak cluster. Do you need any help?

AWS OpsThe Riak on AWS slide

Instances & I/O

• Riak’s I/O patterns

• Small blobs from many places on disk

• EBS best at bulk reads & writes

Instance types

•m1.large & m1.xlarge

• EBS-optimized

•Consider PIOPS

•High I/O Quadruple XL (hi1.4xlarge) - SSDs!

NEW!

NEW!

NEW!

More instances to compensate for the performance variability of virtualized resources

Prepare Yourself

Cluster Sizes

• 5 nodes or moreBest performance & growth

• Scales linearly

•Don’t be stingy

Ring Size

•Change default value!

•Cannot be reset

• Ring size: 2568/64 vnodes per node

•Don’t go above 512

Provisioning

App

haproxy

MyRiakCluster: node-0

MyRiakCluster: node-1

MyRiakCluster: node-2

MyRiakCluster: node-3

MyRiakCluster: node-4

Riak Cluster

HTTPor

ProtobuffApp

haproxy

App

haproxy

Disable Swap• Riak process pages should

not hit swap

• Swap usage can result in unresponsive server

• Let the kernel kill it

Mount & Scheduler

•mount with noatime flag

•Use deadline scheduler on EBS cat /sys/block/xvdj1/queue/schedulernoop [deadline] cfq

echo deadline > /sys/block/xvdj1/queue/scheduler

Linux Filesystems

• xfs, ext3

• ext4

• barrier = 0

• data = writeback

• ZFS not recommended

Backend choices

•Bitcask*

•help mitigate EBS

• LevelDB

•MemoryCan be

changed and

optimized!

Kernel Optimizations

• /etc/sysctl.conf vm.swappiness = 0net.core.somaxconn= 4000net.ipv4.tcp_timestamps = 0net.ipv4.tcp_window_scaling = 1net.ipv4.tcp_max_syn_backlog = 40000net.ipv4.tcp_sack = 1net.ipv4.tcp_fin_timeout = 15net.ipv4.tcp_keepalive_intvl = 30net.ipv4.tcp_tw_reuse = 1

http://wiki.basho.com/Linux-Performance-Tuning.html

Monitoring

• stats or riak-admin status

•Collectd & Nagios

• Riaknostic

• Riak Control (maybe soon)

Backups

• Backup both the ring and data directories

• Integrate with EY tools

• scp or rsync data directory

• EBS snapshots

Scaling Riak

•Vertical

•Horizontal

• reduces node load

• batch is better

riak-admin cluster replace <old> <new>

Benchmarks

Tests

• 5-node cluster / 5 hours

• 50 concurrent connections

• gets 10 /puts 5 /deletes 1

• Bitcask backend

• Protobuffs protocol

Medium - PIOPS 500

Medium - PIOPS 1K

Medium - Ephemeral

FAIL!

XL - EBS

XL -PIOPS 500

4XL - EBS

4XL - PIOPS 1k

What’s next?