Operations & Tuning

Data Model First

Operations Can’t Fix a Bad Data Model

Sizing for Latency

CPU

Memory

Disk

Sizing for Latency

CPU

Buffer Cache

Disk

JVM

Sizing for Operations

(C * Nodes * Cores) / Replication Factor = Ops/Second

C is a hardware constant!• 3000 AWS Non-SSD!• 4000 Commodity SAS Hardware!• 12000 Commodity SSD Hardware

Sizing for Latency

•30-130 microseconds in memory!•100 microseconds to 12 milliseconds for SSD!•9 milliseconds to 60 milliseconds for mechanical disks

Shared Storage

DO NOT USE SHARED STORAGE!!!!!

Disable Access Time

/dev/xvdb /data1 auto defaults,noatime,nodiratime 0 2

Warm The Buffer Cache

➜ ~ find /var/lib/cassandra -name '*.db' -exec cat {} > /dev/null \;

Disable NUMA Zone Reclaim

➜ ~ echo 0 > /proc/sys/vm/zone_reclaim_mode

Use SSD

25% more expensive!10x Faster

Tuning SSD: Read Ahead

➜ ~ sudo blockdev --report RO RA SSZ BSZ StartSec Size Devicerw 256 512 4096 0 6442450944 /dev/xvdarw 256 512 4096 1920 6144000000 /dev/xvda1rw 8 512 4096 0 343568023552 /dev/xvdbrw 256 512 4096 0 343568023552 /dev/xvdc

➜ ~ sudo blockdev --setra 8 /dev/xvdb

Tuning SSD: Scheduler

➜ ~ echo 0 > /sys/block/xvdb/queue/rotational➜ ~ echo noop > /sys/block/xvdb/queue/scheduler

Compaction

Mechanical Disk = Size Tiered!SSD = Leveled Compaction

TTL

TTL Compaction!https://issues.apache.org/jira/browse/CASSANDRA-5228!!

Default TTL!https://issues.apache.org/jira/browse/CASSANDRA-3974

Cassandra Heap Settings

8G MAX_HEAP !2G NEW_HEAP

Solr/Cassandra Heap Settings

14G MAX_HEAP !2G NEW_HEAP

Hadoop/Cassandra Heap Settings

10G MAX_HEAP !2G NEW_HEAP

Kernel

Modern Kernel = 30% performance boost

Monitoring

nodetool cfhistograms

Offset SSTables Write Latency Read Latency Row Size Column Count 1 3579 0 0 0 0 2 0 0 0 0 0 . . . 35 0 0 0 0 0 42 0 0 27 0 0 50 0 0 187 0 0 60 0 10 460 0 0 72 0 200 689 0 0 86 0 663 552 0 0 103 0 796 367 0 0 124 0 297 736 0 0 149 0 265 243 0 0 179 0 460 263 0 0 . . . 25109160 0 0 0 0 0

nodetool tpstatsPool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 15 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 0 0 3674 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 0 0 0 MemoryMeter 0 0 1 0 0 MemtablePostFlusher 0 0 267 0 0 FlushWriter 0 0 9 0 1 MiscStage 0 0 0 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 !Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0

Warning Signs• Pending compactions > 15!• Parnew collections > 1 second!• CMS collections > 5 minutes!• Disk utilization above 40% or 70%!• CFHistograms write throughput > 150 μs!• CFHistograms read throughput > 50 ms!• Really large rows in CFHistograms!• Anything blocked in TPStats

Tools