Lot’s of Tools, Few Solutions
● No Master Blueprints or Distributions
● Docker is Not Enough
● Docker Hub is a Mess
● Fleet has Obvious Limitations
● Managing Systemd Units is Cumbersome
● Logging is a Real Pain
● Data Persistence is Necessary
● Kubernetes & Mesos are Overkill *
But You Run Fat Stacks...
Redis, Memcache, Varnish, MongoDB,
CouchDB, Storm, Jenkins, NodeJS, MySQL,
Postgres, Apache, Nginx, Tomcat, Mongrel,
Mongrel2, Sinatra, Riak, Graphite, Cassandra,
Hadoop, ActiveMQ, Solr, RabbitMQ,
Elastic Search, Logstash, Kibana,
HAProxy, Postfix, and Graphite
What Not To Do
☒Manually Create Unit Files
☒ Blindly Use Docker Hub Images
☒ Run Data Volumes without a Backup Plan
☒Manual Configuration or Conf Management
☒Modify CoreOS Outside of Cloud Init
☒ Execute Multiple Processes per Container
☒ Expose Lots of Public Ports
Docker Madness
Ehh?? What am I looking at?68ce1cebbf07 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10
days 80/tcp tp_web-2
3f26d14fe567 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10
days tp_web_backups
a77bcb74d75c 741583d7dbd1 "/usr/sbin/apache2ct 10 days ago Up 10
days 80/tcp tp_web-1
98f15fac74ca 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10
days tp_mysql_backups
8019d38dd49b cloudposse/library:btsync "\"/bin/sh -c '/usr/ 10 days ago Up 10
days 44444/tcp tp_btsync
652e815800d7 cloudposse/library:mysql "/entrypoint.sh mysq 10 days ago Up 10
days 3306/tcp tp_mysql
21a18d390135 0eeeca18f709 "/bin/bash /sync.sh 10 days ago Up 10
days vps_akira_backups
751f682f9606 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10
days 0.0.0.0:30000->22/tcp vps_osterman
3da29f6eda58 1c3ff222069c "/bin/sh -c 'bootstr 10 days ago Up 10
days 0.0.0.0:30002->22/tcp vps_akira
02c
Fleet-ing Commands
WTF Fleet is Not much better…[email protected] 3ea56e45.../167.114.0.38 active
running
[email protected] 3ea56e45.../167.114.0.38 active
running
osterman_web_announce.service 3ea56e45.../167.114.0.38 active
exited
osterman_web_backups.service 3ea56e45.../167.114.0.38 inactive dead
registrator_skydns.service 3ea56e45.../167.114.0.38 active
running
tp_btsync.service 3ea56e45.../167.114.0.38 active
running
tp_mysql.service 3ea56e45.../167.114.0.38 active
running
[email protected] 3ea56e45.../167.114.0.38 active
running
[email protected] 3ea56e45.../167.114.0.38 active
running
vps_akira.service 3ea56e45.../167.114.0.38 active
running
vps_akira_backups.service 3ea56e45.../167.114.0.38 active
running
Sample Apache Systemd Unit File
[Unit]
Description=Apache Web Server
After=docker.service
Requires=docker.service
[Service]
User=core
EnvironmentFile=/etc/environment
TimeoutStartSec=0
ExecStartPre=-/usr/bin/docker kill ib_web-1
ExecStartPre=-/usr/bin/docker rm ib_web-1
ExecStartPre=/usr/bin/docker --debug=true pull \
registry.hub.docker.com/cloudposse/library:apache-php
(Continued)
ExecStart=/usr/bin/docker run --name ib_web-1 \
--rm \
--dns=${COREOS_PRIVATE_IPV4} \
--dns-search=ib.cloudposse.local \
--volume /media/sdb/ib_web:/var/www/html \
-e "DB_USER=wordpress" \
-e "DB_PASS=XXXXXXXX" \
-e "DB_HOST=web.mysql.ib.cloudposse.local" \
-e "DB_NAME=wordpress" \
-e "SERVICE_NAME=ib/web" \
-e "SERVICE_ID=1" \
registry.hub.docker.com/cloudposse/library:apache-php
ExecStop=-/usr/bin/docker kill ib_web-1
ExecStop=-/usr/bin/docker rm ib_web-1
The Missing Pieces
♟Fleet Orchestration
♟Systemd Unit Templates
♟Basic Building Blocks
♟Naming Conventions
♟ Introspection
♟Overlay Networking
What if I told you….
There was a way you could quickly spin up
docker containers for everything, running
coreos in the cloud or bare metal, with
multiple environments or stages, using
simple commands on your laptop
it would just work? =)
The Blue Print
● Ensemble Services Distribution✓ 60+ Essential Services (e.g nginx, varnish, mysql)
✓ HTTP Router with Vulcand
✓ RFC1918 Overlay Network
✓ Service Composition, Orchestration
✓ Library Unit Templates, Images
✓ DNS Based Service Discovery & Registration
✓ Logging to PaperTrail, DataDogHQ, LogEntries
✓ Remote VPN Access
✓ Backups & BitTorrent Data Replication
Ensemble Command Line
Usage: bin/ensemble options [action] [service] [unit]
Some of the Supported actions:
build Compile the m4 templates for each service
commit Execute "git commit" modified units for each service
push Execute "git push" to master committed units for each service
kickstart Shortcut for stop+destroy+start specified units
restart Shortcut for stop+start specified units
view Display the current unit definition
template Display the template definition
status Display status of all units in cluster
stats Display realtime performance metrics of specified units in cluster
journal Stream stdout logs for specified unit from each server in the cluster
shell Attach to the container and execute a shell (/bin/bash)
Actions can be combined using +
Command Line Examples
# Build all configurations for all services
ensemble --config deployment.yml build
# Build and deploy only the “ib” service
ensemble --config deployment.yml build+kickstart ib
# Give me the operating status of the “ib” service
ensemble --config deployment.yml status ib
# Open up a shell inside the web container
ensemble --config deployment.yml shell ib web@1
# Display real-time memory and network utilization of the “ib” service
ensemble --config deployment.yml stats ib
# Commit changed configurations and push them to master
ensemble --config deployment.yml commit+push ib
Tons of Ready Made Templates
datadog.service.m4, logentries.service.m4, logspout.service.m4,
boundary_agent.service.m4, newrelic_sysmond.service.m4,
nodejs.service.m4, apache.service.m4, openvpn.service.m4,
docker_registry.service.m4 duplicity.service.m4,
haproxy_with_discover.service.m4, os_swap.service.m4
jenkins_master.service.m4, jenkins_swarm.service.m4,
kiwiirc.service.m4, memcache.service.m4, skydns.service.m4
mysqldump.service.m4, mysql_galera.service.m4, generic.timer.m4,
mysql_galera_cluster.service.m4, mysql_galera_garbd.service.m4,
mysql_galera_lb.service.m4, mysql_server.service.m4,
btsync.service.m4, postfix.service.m4, redis.service.m4,
registrator.service.m4, vps.service.m4, vulcand.service.m4,
os_mount.service.m4
Ensemble Configuration (YAML)fleet_tunnel: "cluster.cloudposse.local" # Cluster to administer
fleet_ssh_timeout: 20 # How long to wait before giving up
template_path: "~/Dev/ensemble/units" # Where to find the templates
template_engine: "m4" # How to rewrite the templates
target_path: "~/Dev/cluster-configs/units" # Where to send the rendered units
globals: # Global variables can be used anywhere in the YAML
config
cluster_name: "cluster-1" # Sample definition
container_storage: "@(mount_dir)/sdb" # Where we’ll store persistent data (it will be backed
up!)
mount_dir: "/media" # Mount point on CoreOS for persistent storage
services: # Services are composed of a collection of units
- include: "services/core.yml" # CoreOS extensions deployed via units
- include: "services/jenkins.yml" # Jenkins CI with Jenkins Swarm
- include: "services/osterman.yml" # My own website and virtuals
- include: "services/tp.yml" # TP Deployment
- include: "services/isc_staging.yml" # Staging environment for ISC
- include: "services/isc_prod.yml" # Production environemnt for ISC
Pretty Service Configurations
# Osterman Associates
name: "osterman" # The name of this service
globals: # Local service globals # Set/Override New Globals
mysql_user: "wordpress"
mysql_pass: "XXXXXXXXXX"
mysql_host: "mysql.@(service).@(local_domain)"
mysql_port: 3306
mysql_database: "wordpress"
mysql_root_pass: "XXXXXXXXXXXXXX"
mysql_backups_volume: "@(container_storage)/@(service)_mysqldump:/backups"
mysql_volume: "@(container_storage)/@(service)_mysql:/var/lib/mysql"
web_volume: "@(container_storage)/@(service)_web:/var/www/hosts"
Minimal Unit Configurations# Database instance for wordpress
- name: "mysql" # Name of this unit
template: "mysql_server.service.m4"
type: "service"
define:
"DOCKER_VOLUME": "@(mysql_volume)"
"DOCKER_IMAGE": "@(docker_registry)/cloudposse/library:mysql"
"MYSQL_USER": "@(mysql_user)"
"MYSQL_PASS": "@(mysql_pass)"
"MYSQL_HOST": "@(mysql_host)"
"MYSQL_DATABASE": "@(mysql_database)"
Clustering Made Easy
# Apache web service for wordpress multisite
- name: "web@%i"
template: "apache.service.m4"
type: "service"
instances: 5
# Define all the virtual hosts referencing another definition
- name: "[email protected]"
from: "vhost@%i"
- name: "[email protected]"
from: "vhost@%i"
Screencast
● Overview
● Sample Configurations (YAML, m4)
● Demonstration of COOL Commands
● Data Dog, Paper Trail, Boundary
BAM! That’s all I got...
Ask Erik Osterman
(415) 535-8615
Top Related