Why does my choice of storage matter with cassandra?

Why does my choice of storage matter with Cassandra? Johnny Miller, Solutions Architect @CyanMiller www.linkedin.com/in/johnnymiller

Quote

“The single biggest predictor of success or failure with a Cassandra deployment is in storage choice”

Patrick McFadin, Chief Evangelist for Cassandra, @PatrickMcFadin

©2014 DataStax Confidential. Do not distribute without consent. 2

Cassandra Storage Engine


Inserts/Updates


Memtables are organized in sorted order by row key and flushed to SSTables sequentially (Read/Write)

Ordered Map of KVP, (Immutable, Read Only)

Append only file structure, providing interim durability for writes before they get flushed to SSTables (Write Only)

Reads


Deletes

•  Unlike most DBs, deleted data is not immediately removed from disk.

•  A marker called a tombstone is written to indicate the the column is deleted

•  A tombstones exist for a configurable period of time, and are only deleted from disk via compaction after that time has expired.

•  Deletes are just as fast as inserts J


Compaction

•  Regular compaction of data in Cassandra is essential for a healthy and performant cluster. •  SSTables are immutable

•  Get rid or duplicate/overwritten data

•  Drop deleted data and tomnstones

•  Data in SSTables is sorted by partition key, the effect of which is that while the SSTables are being consolidated, the disk I/O is not random.


Compaction Strategies

•  There is a choice of three strategies Cassandra can use for compaction and all have different disk I/O profiles and capacity requirements.

•  SizeTieredCompactionStrategy (default) •  Using this strategy causes bursts in I/O activity while a compaction is in process

•  These I/O bursts can negatively affect read-heavy workloads, but typically do not impact write performance.

•  Data highly likely to be spread across multiple SSTables i.e. multiple disk seeks

•  LeveledCompactionStrategy •  ~90% of the time, data will be in only a single SSTable i.e. minimal disk seeks

•  However, there is significantly higher Disk I/O than size tiered compaction in order to guarantee how many SSTables data may be spread across

•  Due to high disk I/O rarely appropriate for on traditional HDD

•  DateTieredCompactionStrategy (C* 2.0.11+ and 2.1.1+) •  Stores data written within a certain period of time in the same SSTable.

•  Can store data that has been set to expire using TTL in an SSTable with other data scheduled to expire at approximately – can just drop the SSTable without any compaction!


Choice of storage matters

•  Most databases rewrite modified data in place and writes are buffered and then flushed to disk as random writes.

•  With Cassandra: •  Disk writes are typically sequential append only

operations •  On-disk tables are written in a sorted order so compaction

running time increases linearly with the amount of data

•  So choice of storage is pretty important!!


Disks and Configuration Options


Quote

“For many applications, we are no longer constrained by hard drive capacity, but by seek speeds.

Essentially, a 7200 RPM hard drive is capable of delivering approximately 100 seeks per second, and this has not changed in more than 10 years, even as disk capacity has been doubling every 18–24 months.

In fact, if you have a big data application which required a half a petabyte of storage, what had previously required 1024 disk drives when we were using 500 GB drives, now that 3 TB disks are available, only 171 disk drives are needed.

So a storage array capable of storing half a petabyte is now capable of 80% fewer seeks.” - Ted Ts’o Maintainer of the ext4 file system in the Linux kernel


Hard Drive/Spinning Disk


This part actually has to move! This bit spins around very fast

So what can we do?

•  Memory? •  Caching can help, but the hit rate has to be extremely high to mitigate the mechanical

latency of spinning disks

•  Get rid of the moving parts! •  Mechanical media will never be able to keep up under load

•  Today’s databases service multiple users with difference access patterns

•  A relatively small number of concurrent disk reads can result in seconds of latency

•  SSDs don’t have moving parts

•  SSDs can eliminate entire classes of problems

•  With Cassandra in particular, you will save a lot of money on staff resources by investing in SSDs up front

•  Compactions can be tough on flash, but it’s not as bad as you think


The best way to do it – SSDs!

•  What is an SSD? •  Solid State Drive •  Bits stored in NAND Flash Memory •  No moving parts •  “Seeks” 2-3 orders of magnitude faster than spinning disk

•  What’s the catch?

•  Smaller capacity •  More expensive •  Flash wears out

In practice, this is not a problem – if it makes you nervous, keep spares


The IO Scheduler •  NOOP - use this scheduler if you know another IO device (like a RAID

card) will be doing its own IO scheduling. The NOOP scheduler is just a pass-through.

•  Deadline - otherwise, use the deadline scheduler

•  Tell the OS the drive is non-rotational

•  Tune read-ahead way down – start at 0 and work your way up

Don’t forget to tune the OS for SSDs


echo deadline > /sys/block/<drive>/queue/scheduler

echo 0 > /sys/block/sda/queue/rotational

blockdev –setra 0 /dev/<drive>

Use SSDs

•  More flexibility and substantial performance benefit •  Typically 10x the performance for less than 2x the cost (potentially lower) when compared

with HDDs. •  You can use LeveledCompactionStrategy •  SSD drives can scale up to cope with larger compaction overheads while simultaneously

serving many random reads. •  Netflix found that they could half the total system cost to achieve the same level of

throughput.

•  Additionally the mean read request latency was reduced from 10ms to 2.2ms and 99th percentile request latency was reduced from 65ms to 10ms.

•  http://techblog.netflix.com/2012/07/benchmarking-high-performance-io-with.html.


The worst way to do it

•  Shared Storage •  Cassandra is a shared-nothing architecture with no single point of failure

•  Adding shared storage adds a single point of failure •  Irrespective of the terrible performance – this alone is enough reason not to do

it.


Shared Storage

Local storage configuration

•  RAID or JBOD? •  RAID – Redundant Array of (Independepent/Inexpensive) Devices •  JBOD – Just a Bunch Of Disks

•  RAID •  Common Cassandra RAID levels are 0, 1, 10 •  RAID-0 is most common, but means all data on node must be rebuilt

from other nodes when a drive fails

•  JBOD •  Drives are listed individually in cassandra.yaml •  Failed drives can be replaces individually


How to choose between RAID or JBOD?

•  Performance •  For SSDs, not so much…

•  Compactions are usually throttled significantly below bus speed

•  So a single SSD usually has sufficient throughput

•  Throughput is really the only advantage RAID buys

•  Manageability

•  Pick the option that best fits the deployment scenario •  If using SSD, and drives can be replaced, choose JBOD •  Otherwise, RAID is probably the right choice.


How to choose between RAID or JBOD?

•  Cloud provider •  EC2 ephemeral SSD can’t be replaced, use RAID (and dont use EBS)

•  GCE persistent SSD volumes can be replaced, JBOD is useful

•  Not all drives are hot swapable •  PCIe devices can’t conveniently be replaced

•  SSD Spares for JBOD mode •  Keep spare SSDs online, but not in use •  Allows the node to be easily brought back online

with a quick config change


Comparison Data


FusionIO ioDrive II


Reads

Writes

Latency (microseconds)

PNY XLR8 SSD (consumer grade MLC)


Samsung 840 Pro SSD (consumer grade MLC)


7200RPM SATA


7200RPM SAS


10K SATA


15K SAS


All Drives


All SSDs


Conclusion

•  Using SSDs is a good idea •  Better response times •  Less variance in performance •  Significantly higher throughput so fewer servers needed


VS

Thank You

We power the big data apps that transform business.

Why does my choice of storage matter with cassandra?

Technology

Transcript of Why does my choice of storage matter with cassandra?