The Last Pickle: Repeatable, Scalable, Reliable, Observable: Cassandra
Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016
Transcript of Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016
![Page 1: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/1.jpg)
Real world tales of repair
![Page 2: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/2.jpg)
CASSANDRA SUMMIT - SEPTEMBER 2016
Alexander Dejanovski@alexanderdeja
Consultantwww.thelastpickle.com
Datastax MVP for Apache Cassandra
Licensed under a Creative Commons Attribution-NonCommercial 3.0 New Zealand License
![Page 3: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/3.jpg)
About The Last Pickle
We help people deliver and improve Apache Cassandra based solutions.
With staff in 5 countries and over 50 years combined experience in Apache Cassandra.
![Page 4: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/4.jpg)
![Page 5: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/5.jpg)
![Page 6: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/6.jpg)
What and why ?Full repair
Incremental repairHow to make it workAutomated repairs
www.thelastpickle.com
![Page 7: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/7.jpg)
What is repair ?
A maintenance operation that (briefly) restores strong consistency throughout the cluster
www.thelastpickle.com
![Page 8: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/8.jpg)
Why do we need repair ?
- Eventual consistency- Downtime / failure recovery- Safe deletes
www.thelastpickle.com
![Page 9: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/9.jpg)
Tombstones need repair too
Missing tombstones can lead to zombie data(repair within gc_grace_seconds)
www.thelastpickle.com
![Page 10: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/10.jpg)
What and why ?Full repair
Incremental repairHow to make it workAutomated repairs
www.thelastpickle.com
![Page 11: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/11.jpg)
How does anti-entropy repair works ?
Reads all data
www.thelastpickle.com
![Page 12: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/12.jpg)
How does anti-entropy repair works ?
Reads all dataCalculates hashes
www.thelastpickle.com
![Page 13: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/13.jpg)
How does anti-entropy repair works ?
Reads all dataCalculates hashesCompares hashes
www.thelastpickle.com
![Page 14: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/14.jpg)
How does anti-entropy repair works ?
Reads all dataCalculates hashesCompares hashes
Streams mismatching partitions
www.thelastpickle.com
![Page 24: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/24.jpg)
Improving repair
repairing each range once is enough
www.thelastpickle.com
![Page 26: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/26.jpg)
Improving repair
nodetool repair -pr
not suitable for node recovery
www.thelastpickle.com
![Page 27: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/27.jpg)
Repair too slow ?
Sequential repair is the default
since C* 2.0
www.thelastpickle.com
![Page 29: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/29.jpg)
The problem with dense nodes
Overstreaming
Leaves of the Merkle tree contain several partitions.
www.thelastpickle.com
![Page 30: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/30.jpg)
The solutions with dense nodes
cassandra_range_repair(Matt Stump & Brian Gallew)
Breaks the repair sessions in n steps
www.thelastpickle.com
![Page 31: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/31.jpg)
The solutions with dense nodes
vnodes : one repair session per vnode
Drawback : if you have many vnodes, repair takes longer
www.thelastpickle.com
![Page 33: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/33.jpg)
The early days of your cluster
Node density is low,repair works just finehowever you run it.
www.thelastpickle.com
![Page 34: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/34.jpg)
The early days of your cluster
So maybe like I did,you run « nodetool repair »on all nodes… at the sametime
www.thelastpickle.com
![Page 35: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/35.jpg)
The (not so) early days of your cluster
As nodes gets higher in density, repair takes longer… and longer…
www.thelastpickle.com
![Page 36: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/36.jpg)
The (not so) early days of your cluster
… and latencies riseas repair is a CPU and I/O intensive operation
www.thelastpickle.com
![Page 37: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/37.jpg)
Your cluster is a grown up now
… until it breaks your cluster
www.thelastpickle.com
![Page 39: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/39.jpg)
How can it break ?
Load gets too highYou don’t meet your latency SLA anymore
www.thelastpickle.com
![Page 41: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/41.jpg)
How can it break ?
Load gets too highStreams get stuck
www.thelastpickle.com
![Page 42: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/42.jpg)
How can it break ?
Load gets too highStreams get stuck
and out of nowhere, all nodes start to eat all your CPU doing nothing
www.thelastpickle.com
![Page 43: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/43.jpg)
The fun part ?
You need to run repairto recover from the repair outage !
www.thelastpickle.com
![Page 44: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/44.jpg)
The cluster keeps growing And you realize orchestration is neededto stop blowing up your cluster
www.thelastpickle.com
![Page 45: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/45.jpg)
Orchestrating repair
Repair must not run on all nodesat the same time
www.thelastpickle.com
![Page 46: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/46.jpg)
Tools to orchestrate repairs
OpsCenter repair service (DSE users)
Spotify reaper
www.thelastpickle.com
![Page 47: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/47.jpg)
Spotify reaper
https://github.com/spotify/cassandra-reaper
www.thelastpickle.com
![Page 49: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/49.jpg)
Spotify reaper
Performs subrange repairLimits repair pressure
www.thelastpickle.com
![Page 50: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/50.jpg)
Spotify reaper
Performs subrange repairLimits repair pressureRetries failed sessions
www.thelastpickle.com
![Page 51: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/51.jpg)
Spotify reaper
Performs subrange repairLimits repair pressureRetries failed sessions
Schedules cyclic repairs
www.thelastpickle.com
![Page 52: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/52.jpg)
Spotify reaper
Performs subrange repairLimits repair pressureRetries failed sessions
Schedules cyclic repairsOptimizes cluster load
www.thelastpickle.com
![Page 53: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/53.jpg)
Spotify reaper - with UI (thx Stefan Podkowinski)
GUI screenshots
www.thelastpickle.com
![Page 54: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/54.jpg)
What and why ?Full repair
Incremental repairHow to make it workAutomated repairs
www.thelastpickle.com
![Page 56: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/56.jpg)
Here comes the savior !
C* 2.1 introduces incremental repairDefault repair mode since C* 2.2
www.thelastpickle.com
![Page 59: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/59.jpg)
Anticompaction (repair on all ranges on local node)
www.thelastpickle.com
![Page 60: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/60.jpg)
Incremental repair looks awesome… …but has flaws and drawbacks
www.thelastpickle.com
![Page 61: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/61.jpg)
Incremental repair caveats
Carefully prepare your switch to incremental repair
www.thelastpickle.com
![Page 62: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/62.jpg)
Incremental repair caveats
Carefully prepare your switch to incremental repair
i.e. do not run « nodetool repair -inc »straight away…
www.thelastpickle.com
![Page 63: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/63.jpg)
Incremental repair caveats
It doesn’t handle missing/corrupted datathat was already repaired
www.thelastpickle.com
![Page 64: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/64.jpg)
Incremental repair caveats
It splits SSTables in 2 setsthat cannot be compacted together
(think tombstone purge)
www.thelastpickle.com
![Page 65: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/65.jpg)
Incremental repair caveats
It is incompatible with subrange repair(anticompaction)
www.thelastpickle.com
![Page 66: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/66.jpg)
Incremental repair caveats
It doesn’t like concurrency very mucheven in C* 3.x
www.thelastpickle.com
![Page 67: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/67.jpg)
Incremental repair caveats
Validator.java:261 - Failed creating a merkle tree for [repair #e4c782d0-11fc-11e6-b616-51a3849870bb on table_v2/table_attributes, [(8835460833482333317,8838777311566358575],(-7300486781514672850,-7298192396576668423], (-959298474675167225,-959177964106074209]]], /10.10.10.33 (see log for details)
www.thelastpickle.com
![Page 68: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/68.jpg)
Incremental repair caveats
CompactionManager.java:1320 - Cannot start multiple repair sessions over the same sstables
www.thelastpickle.com
![Page 69: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/69.jpg)
Incremental repair caveats
CASSANDRA-8316
A running anticompation prevents validation compaction
www.thelastpickle.com
![Page 70: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/70.jpg)
Incremental repair caveats
Do not use -pr with incremental repair
www.thelastpickle.com
![Page 71: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/71.jpg)
Incremental repair caveats
Do not use -pr with incremental repair
Useless : data is repaired once only
www.thelastpickle.com
![Page 72: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/72.jpg)
Incremental repair caveats
Do not use -pr with incremental repair
Useless : data is repaired once onlyExpensive : anticompaction overhead
www.thelastpickle.com
![Page 73: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/73.jpg)
Incremental repair will not…
Fix a poor repair strategy
www.thelastpickle.com
![Page 74: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/74.jpg)
Incremental repair will not…
Prevent you from having to run full repair
www.thelastpickle.com
![Page 75: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/75.jpg)
Reaper does not support incremental repair
But this fork does : https://github.com/adejanovski/cassandra-
reaper/tree/inc-repair-that-works
www.thelastpickle.com
![Page 76: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/76.jpg)
Reaper does not support incremental repair
And this one embeds the modded UI : https://github.com/adejanovski/cassandra-
reaper/tree/inc-repair-support-with-ui
www.thelastpickle.com
![Page 77: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/77.jpg)
Reaper does not support incremental repair
Not enough time to write those urls ?
github.com/adejanovski
www.thelastpickle.com
![Page 79: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/79.jpg)
Reaper inc repair fork
No subrange repairSingle repair thread => no concurrency
www.thelastpickle.com
![Page 80: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/80.jpg)
What and why ?Full repair
Incremental repairHow to make it work
Automated repairs
www.thelastpickle.com
![Page 81: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/81.jpg)
Repair best practices
Put your repair strategy in place on day 1
www.thelastpickle.com
![Page 82: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/82.jpg)
Repair best practices
Use appropriate tooling or build your own
www.thelastpickle.com
![Page 83: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/83.jpg)
Repair best practices
Spread repair over a gc_grace_seconds cycle
www.thelastpickle.com
![Page 84: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/84.jpg)
Repair best practices
Adjust repair pressure on your cluster(Reaper does that)
www.thelastpickle.com
![Page 85: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/85.jpg)
Repair best practices
Don’t repair everything !
Pick tables with deletes and those with critical data
www.thelastpickle.com
![Page 86: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/86.jpg)
Repair best practices
If every data is critical, then none is ;)
www.thelastpickle.com
![Page 87: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/87.jpg)
Repair best practices
Be tight on your schedule with inc repair
Tombstones and anticompaction
www.thelastpickle.com
![Page 88: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/88.jpg)
Repair best practices
Avoid concurrency with inc repairOne node at a time
www.thelastpickle.com
![Page 89: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/89.jpg)
What and why ?Full repair
Incremental repairHow to make it workAutomated repairs
www.thelastpickle.com
![Page 90: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/90.jpg)
The bright future ?
Not having to think about cyclic maintenance repairs
www.thelastpickle.com
![Page 93: Real World Tales of Repair (Alexander Dejanovski, The Last Pickle) | Cassandra Summit 2016](https://reader036.fdocuments.us/reader036/viewer/2022070603/586f75d71a28ab10258b61e1/html5/thumbnails/93.jpg)
Thanks!@alexanderdeja