Mo Servers, Mo ProblemsMo Servers, Mo Problems

Really, containers vs. VMs

What is a problem?

How using containers instead of VMs can help you increase uptime and decrease problems requiring human intervention and decision-making

We are living in the future

We will get our handsdirty

Chapter 1

In which you begin to believe me

when I tell you, “Mo’ Servers Mo

Problems”

Let’s bundle optimized hosting along with amazing workflow tools, team management, and runtime analytics, stick it on a VM, and charge $$$!

Bro!

We did it!300 Clients300 Virtual Machines300 Problems

● We can now support Freemium!● Each site has 3+ environments● Containers for PHP-FPM, Nginx, Mount processes, MySQL DB and Redis● To 300 30GB VMs, 100,000 LAMP stacks, ~750,000 containers (TODO count?)

Problems = Infrastructrue*Sites

PITA Coefficient (O)

PITA Coefficient (O)

ODrupal Developer ~= 0.27

OPage View ~= 2.5e-7

OContainer ~= 0.005

ODrupal User ~= 0.025

OVM/Server ~= 8.3

With two Containers on one VMRisk = ½Likelihood * 2xConsequences

With two, single-tenant VMsRisk = 2xLikelihood * ½Consequences

Risk = Likelihood * Consequence

Self-healing Problems

Problems Requiring Basic Manual Intervention

Problems Requiring Decisions

Problems Requiring Coding

Problems Requiring Hard Decisions

Easy Hard

Humans Decisions Compound

5 servers means 10 (network) problems6 servers means 15 (network) problems

1 more server bought you 5 problems

http://aphyr.com/posts/288-the-network-is-reliable

O(N2) Network Failure Paths

If you want fewer Problems

● Increase Mean Time Between Failure○ You could get more reliable things….where?○ You can get fewer things!

● Decrease Mean Time To Resolution○ You can speed-up detection, insight, resolution○ You can reduce reliance on human decisions

“Chief Chirpa Sucks”

[nick@endpoint9a71a1ef ]$

vs.

[nick@ChiefChirpa ~]$

Chapter 2

In which we use English to describe

WTF containers are, and why people

might want to use them.

This is what our marketers say we built

Resource-constrained, system-isolated, metered processes.

Containers are simply....

Time to container$: systemd-nspawn -D /srv/debian/ date

Spawning namespace container on /srv/debian.

Init process in the container running as PID 9159.

Tue Jun 3 17:32:14 UTC 2014

real 0m0.007suser 0m0.001s

real 0m0.007s

Even if you just run one server...

OS Upgrades SuckCloud VMs get ‘weird’Container migration FTW.

End of lifeis a

way of life!

OS upgrade dropsavg server life

Container Migration to MariaDB

One-click migration to convert thousands of MySQL containers to MariaDB

Chapter 3

In which we plumb the depths of the

/proc filesystem, in search of clues

about CGroups and namespaces

Containersare based on the

CGroups and Namespacesfunctionality on the Linux kernel

cgroups is merely a hierarchy ofprocesses All processes

Development processes

PHP-FPM Drush

Production processes

Drush Rsync

75% 25%

cgroups is merely a hierarchy ofprocesses All processes

Processes for people I don’t like

PHP-FPM Drush

Processes forpeople I like

Drush Rsync

2%98%

cgroups submodules aka Controllers

● memory: Memory controller● cpuset: CPU set controller● cpuacct: CPU accounting controller● cpu: CPU scheduler controller● devices: Devices controller● blkio: I/O controller for block devices● net_cls: Network Class controller● ...

Kernel Interaction: /proc, /sys/fs # Inspect ip forwarding setting

$: cat /proc/sys/net/ipv4/ip_forward

# Turn ip forwarding off/on

$: echo "0" > /proc/sys/net/ipv4/ip_forward

$: echo "1" > /proc/sys/net/ipv4/ip_forward

# Examine file descriptors used by nginx..

$: ls -l /proc/$NGINX_PID/fd/

lrwx------ 1 root Jun 3 13:48 0 -> /dev/null

lrwx------ 1 root Jun 3 13:48 10 -> socket:[64376]

l-wx------ 1 root Jun 3 13:48 2 -> /var/log/nginx-access.log

# Nuke logs

$: rm -rf /var/log/nginx-access.log

# Read log (even after you rm -rf’d it!)

$: tail /proc/$NGINX_PID/fd/2

62.211.78.166 - - [05/May/2014:10:00:54 +0000] "GET /vtiger.php

Kernel Interaction: /proc, /sys/fs

# Create a Control Group named “AA”

$: mkdir /sys/fs/cgroup/memory/AA

# New directory magically contains...

$: ls /sys/fs/cgroup/memory/AA

cgroup.clone_children

memory.kmem.usage_in_bytes memory.limit_in_bytes

cgroup.procs memory.max_usage_in_bytes … ...

Managing cgroups: manually

# Limit AA’s memory to 100 bytes

$: echo 100 > /sys/fs/cgroup/cpu/AA/memory.limit_in_bytes

Managing cgroups: manually

Creating cgroups: libcgroups# Create a Control Group named “AA”

$: cgcreate -g cpu:AA

# Set the ‘cpu.shares’ to 100 for “AA”

$: cgset -r cpu.shares=100 AA

# Run a python script in the “AA” control group

$: cgexec -g cpu:AA python test.py

# Limit teensy’s memory to 100 bytes

$: cgcreate -g memory:teensy

$: cgset -r memory.limit_in_bytes=100 teensy

# Associate current shell’s PID with “teensy”

$: echo $$ > /sys/fs/cgroup/memory/teensy/tasks

# Any command will exhaust memory

$: ls

Killed

memory.limit_in_bytes in action

cpu.shares in action

PID USER PR NI VIRT RES SHR S %CPU 9693 root 20 0 107908 624 532 R 60.08 9692 root 20 0 107908 624 532 R 6.307

cpu.shares = 100

cpu.shares = 10

# Run script within each cgroup

$: cgexec -g cpu:AA python test.py &

$: cgexec -g cpu:BB python test.py &

$: top

● Mount● IPC● PID● User● UTS● Network

Kernel Namespaces

“Before one can share, one must first unshare” - Share Bear

# Run a shell with isolated

# network namespace:

$: unshare --net /bin/bash

Chapter 4

In which we agree that nobody (here)

wants to care about /proc, /sys/fs,

and we investigate alternatives

Container Managers

https://github.com/containers/container-rfc

LXC

● The liblxc library● Several language bindings (python3, lua,

ruby and Go)● A set of standard tools to control the

containers● Container templates

Let Me Contain That For You (lmctfy)

● Created by Google● Open Source(ish)● Every process at Google runs within lmctfy● Supports nested containers

systemd-nspawn

● From systemd project “PID EINS!”● Will ship with all Fedora, RHEL, Ubuntu1

[1] It will ship even with you on boardhttps://speakerdeck.com/joemiller/systemd-for-sysadmins-what-to-expect-from-your-new-service-overlord

# Launch Vagrant

$: vagrant ssh

# Install a base debian tree

$: debootstrap unstable /srv/debian/

# Launch a debian container

$: systemd-nspawn -D /srv/debian/

systemd-nspawn

Container Inception

# Launch a read-only debian container

$: systemd-nspawn --read-only -D /srv/debian/

systemd-nspawn

Docker“In its early age, the dotCloud platform used plain LXC (Linux Containers)....The platform evolved, bearing less and less similarity with usual Linux Containers.”1

[1] http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part

[2] https://prague2013.drupal.org/session/automate-drupal-deployments-linux-containers-docker-and-vagrant

Check out @ricardoamaro’s Drupalcon Prague session2

Containerizeralater SpectrumDocker nspawn lxc lmctfy

And once you get containers….

http://coreos.com/blog/cluster-level-container-orchestration/

● Servers solve and create problems● Containers yield agile portability● Containers = CGroups + namespaces● Use tools to manage containers● The future is now

Pantheon, a platform for the content web, running 10s of Ks of LAMP CMS installshttps://www.getpantheon.com/customers IMAGES

Thanks!Nick Stielau@nstielaugithub.com/nstielau/containerz

Image CreditsContainers: https://flic.kr/p/4o3Ria

Clouds: https://flic.kr/p/hHRdBL

Back to the Future (Lego): https://flic.kr/p/fbThy5

Dirty Hands: https://flic.kr/p/8G3aM5

Risk: https://flic.kr/p/81nfaV

Pita Equation: http://www.codecogs.com/latex/eqneditor.php

Pita Evil Eyes: http://www.clipartbest.com/cliparts/7ia/4eL/7ia4eL9iA.png

Containers http://bighugelabs.com/onblack.php?id=6764705137&size=large

CGroups http://fbcg.com/small-groups/

Pengiun Container: http://2.bp.blogspot.com/-47sakFH6uSw/UXgrhNqYF8I/AAAAAAAAHzQ/0W8zFVgR--w/s1600/lxc.png

No Logo: http://static.tumblr.com/i4bgb5d/Uzblps3wo/no-logo-1.jpg

Book sprectrum: https://flic.kr/p/k5jmja

Bottles: https://flic.kr/p/nj8jMn

Mac: https://flic.kr/p/auKEX2

Corn: https://flic.kr/p/6NVL68