10/17/2013 - Docker @ RelateIQ - Scott Bessler & John Fiedler

Docker @ RelateIQ

@

Scott Bessler scott@relateiq.com@scottbessler

File Repohttps://github.com/relateiq/docker_public

John Fiedlerjohn@relateiq.com@johnfiedler

Blogblog.relateiq.com

RelateIQ Twitter@relateiq

Agenda

● Part 1 - What we did with Docker● Part 2 - Why we want to replace Chef● Part 3 - Future plans using Docker

Part 1. Things were getting out of control... Bring on hack day!

Hack Day!

+

+ =

=

What we needed to build

Kafka

Storm

Redis

Cassandra

MongoDB

Zookeeper

Jetty

Nginx

+ Private Docker Registry

Min

imum

Cru

shin

g IT

Docker Files

Docker Files

Elasticsearch

End product

$ devenv start

$ devenv-inner start

Kafka

Redis

Cassandra

MongoDB

Zookeeper

ubuntu

MAC OS X

OneCommand!

VMContainers

Elasticsearch

Orchestration scripts

● Controls Vagrant● Controls inner script

devenv.sh (outer) [up|update|ssh]

devenv-inner.sh (inner) [stop|start|kill|update|status|restart]

start(){

mkdir -p $APPS/zookeeper/data

mkdir -p $APPS/zookeeper/logs

ZOOKEEPER=$(docker run \

-d \

-p 2181:2181 \

-v $APPS/zookeeper/logs:/logs

server:4444/zookeeper)

echo "Started ZOOKEEPER in

container $ZOOKEEPER"

echo "Wiring containers together… "

(later slide)kill/stop(){

echo "Killing all docker containers:"

docker ps | tail -n +2 |cut -d ' ' -f 1 | xargs docker kill

}

update(){

apt-get update

apt-get install -y lxc-docker

docker pull server:4444/zookeeper

docker pull server:4444/redis

docker pull server:4444/cassandra

docker pull server:4444/elasticsearch

docker pull server:4444/mongo

docker pull server:4444/kafka

}

● Controls Docker containers

Dockerfile’s

FROM server:4444/oracle-java7

RUN apt-get update

RUN apt-get install -y git curl build-essential make gcc wget

RUN cd /opt && wget http://apache.mirrors.pair.com/cassandra/1.2.9/apache-

cassandra-1.2.9-bin.tar.gz

RUN cd /opt && tar zxf apache-cassandra-*.tar.gz

RUN rm /opt/*.tar.gz

RUN mv /opt/apache-cassandra-* /opt/cassandra

RUN apt-get install -y lsof

ADD cassandra.yaml /opt/cassandra/conf/cassandra.yaml

ADD cassandra-env.sh /opt/cassandra/conf/cassandra-env.sh

ADD log4j-server.properties /opt/cassandra/conf/log4j-server.properties

ADD start.sh /opt/cassandra/bin/start.sh

ADD cassandra-topology.properties /opt/cassandra/conf/cassandra-topology.

properties

RUN chmod 755 /opt/cassandra/bin/start.sh

RUN mkdir /logs

...

VOLUME [ "/logs" ]

...

EXPOSE 7000

….

CMD "/opt/cassandra/bin/start.sh"

#!/bin/bash

echo "Cassandra node

configuration:"

echo $CASS_SEEDS

echo $CASS_TOKEN

echo $CASS_LOCAL_IP

HOST=`hostname`

echo "127.0.0.1 $HOST" >>

/etc/hosts

/opt/cassandra/bin/cassandra -fstar

t.sh

Std

Imag

eDockerfile

Networking - port forwarding

Question:Dude, where’s mongo?

Answer:Port forwarding

Example:> Docker file

expose 9999 (dockerfile)> Docker

docker run -p 8000:9999> VAGRANT

config.vm.forward_port 27018, 8000> localhost:8000

Tip:Always use the same port if you can

MongoDB

dockerrun -p 9999:27018

Machine

Containers

localhost:27019

vagrantconfig.vm.forward_port 9999,27019

docker fileexpose 27018

Networking - connecting two containers

KafkaZookeeper

192.168.1.1 192.168.1.2

docker

Machine

Containers

vagrant

pipework br1

Question:Kafka: Dude, where’s Zookeeper?

Answer:Pipework https://github.com/jpetazzo/pipework

Example:> startup the container

KAFKA = docker run -e zooip=192.168.1.1> Pipework

pipework br1 $KAFKA 192.168.1.2 pipework br1 $ZOOKEEPER 192.168.1.1 (dockerfile)

Tip: Use this for clustering

Best practices

● Data● Logs● Don’t end with tailing ● End with foreground

execution● Start script for runtime

configuration● 42 layers, combine

Dockerfile lines● Up Vagrant RAM default

KafkaMongoDB

VM

*Start Script*Foreground Execution*42 layers only

/logs/mongo/data/mongo

/logs/kafka/data/kafka

Machine

Containers

What we found *Abstraction

*Standard Data and log dirs*Static IP

Chef

4 reasons why we want to replace Chef with Docker in prod

Docker Dockerfile● Admittedly less powerful● What you see is what you

get● Environment variables● Inspect environment at

any step

1. Dynamic Configuration Is Too Complex

Chef● Dynamic configuration

through complex attribute system

Docker● Once it’s in, it’s in.● Even if you change an

image, only incremental changes need to be sent to hosts

Chef● Dependencies are

external forever● Slower. You pay the price

on every node deployed● Under load? Want a new

node? Uh oh, the file download is missing.

● Node creation needs to be foolproof

● Or just use chef-solo to bake images

2. External Dependencies Cause Flaky Provisioning

Chef● Configuration change? Try

to apply it to running node● Which changes restart

which services?● Removed something?

○ knife ssh○ tombstone

● Another reason to just bake images

Docker● Ship configuration

changes as image deltas● Nearly instant restarts● Easier to be disciplined

and have each node be identical

3. Configuration Changes Create an Inconsistent State

Dockerfile● Typo? Fix it and start from

the previous successful command

● Containers encourage single-purpose instances (put your monitoring on the host)

Chef● Error? Throw it all out.● Every iteration/test takes:

○ Boot time○ Every step

● Can’t test every host● Can’t test every

combination of cookbooks○ Monitoring○ Logging

4. Developers! Developers! Developers!

Part 3. Whats next? What do we want?

What’s Next for Docker @ RelateIQ?● Replace Chef search (simple node database in

elasticsearch)● Monitoring via StatsD and/or Datadog

Wish List● Dockerfile repo/trusted builds (mentioned on docker-

dev)● Centralized Docker host management● Mac host (no more virtualbox!)

We’re hiring! https://www.relateiq.com/jobs.html