DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Cassandra Summit 2016
Cassandra and Docker Lessons Learned
-
Upload
planet-cassandra -
Category
Engineering
-
view
2.775 -
download
2
Transcript of Cassandra and Docker Lessons Learned
Cassandra and DockerTwo buzzwords or a match made in heaven?
instaclustr.com @Instaclustr
Who am I and what do I do?• Ben Bromhead
• Co-founder and CTO of Instaclustr -> www.instaclustr.com
<sales>
• Instaclustr provides Cassandra-as-a-Service in the cloud.
• Currently in AWS, Azure and Google Cloud in private beta with more to come.
• We currently manage 50+ nodes for various customers, who do various things with it.
</sales>
Objectives
• A quick intro on docker.
• Why docker matters and how it works.
• Working with Cassandra and docker.
• Running C* in a constrained env w/ docker
The ChallengeStatic website
Web frontend
DBQueue
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
Java + Cassandra + Spark
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptop
Production Servers
Mul$p
licity
*of*S
tacks*
Mul$p
licity
*of*
hardware*
environm
ents* Production Cluster
Customer Data Center
Do*services*and*apps*interact*
appropriately?*
Can*I*migrate*
smoothly*and*quickly?*
Static website
Web frontend
Background workers
DB
Analytics
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster Public Cloud Contributor’
s laptopCustomer
Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Dependency madness
Pre 1960’s transportMul$p
licity
*of*G
oods*
Mul$p
ilicity*of*
metho
ds*fo
r*tran
spor$n
g/storing*
Do*I*worry*about*
how*goods*interact*
(e.g.*coffee*beans*next*to*spices)*
Can*I*transport*quickly*and*sm
oothly*(e.g.*from
*boat*to*train*to*truck)*
Also a dependency mess? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Solution: Intermodal Shipping containersMul$p
licity
*of*G
oods*
Mul$p
licity
*of*
metho
ds*fo
r*transpor$n
g/storing*
Do*I*worry*about*
how*goods*interact*
(e.g.*coffee*beans*next*to*spices)*
Can*I*transport*quickly*and*sm
oothly*(e.g.*from
*boat*to*train*to*truck)*
…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another!
A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.!
Docker, shipping containers for codeStatic website Web frontend User DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’s laptop
Mul$p
licity
*of*Stacks*
Mul$p
licity
*of*
hardware*
environm
ents*
Production ClusterCustomer Data Center
Do*services*and*apps*interact*
appropriately?*
Can*I*migrate*
smoothly*and*quickly*
…that can be manipulated using standard operations and run consistently on virtually any hardware platform !
An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…!
Solves the deployment dependency matrixStatic website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster Public Cloud Contributor’
s laptopCustomer
Servers
Why docker matters• Finally Developers have a solution to build once and deploy
anywhere
• Finally Ops/Admin has a solution to configure anywhere
• Finally DevOps is easy
• Dev == Test == Staging == Production
• Move with speed
Docker, how it works.• Runs anywhere (Linux kernel 2.6.32+)
• Uses lightweight VMs:
• Own process space (namespace)
• Process isolation and resource control (cgroups)
• Own network adapter
• Own filesystem (chroot)
• Linux Analog to Solaris Zones, *BSD jails
Docker, how it works.• Difference between a container and a VM
Virtual Machine Container
Docker, how it works.• What about the packaging component?
• Uses Union filesystem to create a git like workflow around your deployed code:
!!
Docker!Container!Image!Registry!
Push%
!!!!
Bins/!Libs!!!!!
App!A!
App!Δ!!
!!!!Bins/!
Docker'Engine' Docker'Engine'
Update'
Host'is'now'running'A’’''
App'Δ''
''''Bins/'
''''
Bins/'Libs'''''
App'A'
''''Bins/'
''''
Bins/'Libs'''''
App'A’’'
Host'running'A'wants'to'upgrade'to'A’’.'Requests'update.'Gets'only'diffs''
Docker is it production ready?
Cassandra and Docker• So how do we get on board the hype train? Without killing
performance or stability?
• Build Cassandra in a docker container, run it, then test.
• Run in dev to get comfortable with it.
• Talk to others who use it in production
• https://github.com/docker/docker/issues - You will spend a lot of time here
Docker + Networking• 1st attempt, throughput dropped in half!
• Writes sucked, streaming sucked, what was going on?
• Quick check with iperf showed a 50% hit in throughput
• Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput.
• Use the host network stack instead (—net=host), only saw a ~10% hit on network performance
Docker + Networking
• Docker uses Linux Ethernet Bridges for basic software defined routing. This will hose your network throughput.
• Use the host network stack instead (—net=host), only saw a ~10% hit on network performance
• Also solves NAT issues in an AWS like networking environment.
Docker + Filesystem• Don’t want to throw it out when you upgrade/stop container.
• Use volume mount folders to the underlying host!
Docker + Filesystem
• The filesystems (AUFS, BTRFS etc) that bring great benefits to Dockers workflow around building and snapshoting containers are not very good for databases.
• UnionFS (AUFS) is terrible for writing lots of big files.
• BTRFS is a pain to use from an ops point of view.
• Hooray volume mounts use the underlying filesystem.
Docker + Process Capabilities
• Mlockall permission denied? A process needs CAP_IPC_LOCK & RLIMIT_MEMLOCK in order to perform this operation. By default docker doesn't assign this to a running container…
• Can’t use native memory. Cassandra becomes slooow.
• Can use --privileged and be done with it. Kind of lazy though
• Use --cap-add instead
Docker + SIGTERM propagation• When stopping the process docker will send a SIGTERM.
• PID 1 does not have default signal handlers!
• Bad if you use a bash script to launch Cassandra
Images shameless copied from https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/
Docker + SIGTERM propagation
• Java to the rescue!
• Make sure you run the cassandra bash script with -f (foreground)
• exec causes the JVM to replace the bash process… making the world a happier place
Docker + SIGTERM propagation• Tools like OpsCenter Server will have trouble with this.
• Can be fixed using a wacky combination of trap and wait stanzas in your OpsCenter Server script (see http://veithen.github.io/2014/11/16/sigterm-propagation.html)
• But now you have a bash script that duplicates init/systemd/supervisord
• The debate rages on…
Docker + CoreOS• Docker + fav OS + CM?, CoreOS + etcd, Swarm + Machine, Deis
etc
• We chose CoreOS (Appeared to be sane, etcd is cool, systemd if you are into that kind of thing)
Docker + CoreOS
• Disable automatic updates + restarts (seriously do this)
• Fix logging, otherwise you will log to 3 locations (/var/log/cassandra, journalctl and dockers json based log
• JVM will exit with error 143 (128 + 15 for SIGTERM). Need to ignore that in your systemd service definition.
Docker + Dev Env
• Docker relies on Linux kernel capabilites… so no native docker in OS X
• We use OSX for dev, so we run vagrant and the CoreOS vagrant file https://github.com/coreos/coreos-vagrant
• Look at https://github.com/tobert/cassandra-docker for something more off the shelf
Docker + C* + Dev Env
• How do I run lots of C* instances on a VM or my dev laptop without it falling over?
• Make it run as slowly, but as stable as possible!
• This is actually a great learning exercise as you discover a lot about how Cassandra works under the hood.
Docker + C* + Dev Env• Set Memory to be super low, edit your cassandra-env.sh:
MAX_HEAP_SIZE="128M" HEAP_NEWSIZE="24M"
Docker + C* + Dev Env• Tune compaction to have free reign and to smash the disk
concurrent_compactors: 1 in_memory_compaction_limit_in_mb: 2 compaction_throughput_mb_per_sec: 0
Docker + C* + Dev Env• Let’s use HSHA thrift server as it reduces the memory per thread
used.
rpc_server_type: hsha
Docker + C* + Dev Env• The HSHA server also lets us limit the number of threads serving in
flight requests, but still have a large number of clients connected.
concurrent_reads: 4 concurrent_writes: 4 rpc_min_threads: 2 rpc_max_threads: 2
• You can play with these to get the right numbers based on how your clients connect, but keep them low.
Docker + C* + Dev Env• This is Dev! Caches have no power here!
key_cache_size_in_mb: 0 reduce_cache_sizes_at: 0 reduce_cache_capacity_to: 0
Docker + C* + Dev Env
• How well does this work?!?!
• Will survive running the insane workload in the c* 2.1 new stresstest tool.
• We run this on AWS t1.micro instances
• Sign up at https://www.instaclustr.com and give our new Developer nodes a spin!
Go forth and conquer!
Questions?