Building containersTurbocharged!

Shopify

• All the commerce

• 165k+ merchants, 8 Billion GMV

• 300M+ unique visits/month

• 4000+ containers

• RoR, MySQL, Redis, Memcache, Elasticsearch, Chef, Go, Podding, Multi-DC,...

Docker at Shopify• Serving production traffic for over a year

• Gives developers more freedom (and responsibility) around how applications are being ran

• Offers super fast and reliable deploys1

1 When it works. Warstories are plenty

Building containers

Dockerfiles

Super simple (Dockerfile is essentially shell commands + a handful of specialized instructions)

Super efficient (Detects changes and skips things that have not changed)

Super inefficient and complicated for anything nontrivial

Simple, eh?FROM ruby:2.1.3RUN apt-get updateRUN apt-get upgradeRUN apt-get install -qq -y libmysqlclient-devRUN useradd --create-home rails_is_funWORKDIR /home/rails_is_funCOPY . /home/rails_is_funRUN bundle installCMD bundle exec unicorn -p 3000

NOPE NOPE NOPE

FROM ruby:2.1.3RUN apt-get updateRUN apt-get upgradeRUN apt-get install -qq -y libmysqlclient-devRUN useradd --create-home rails_is_fun

WORKDIR /home/rails_is_funCOPY Gemfile /home/rails_is_fun/GemfileCOPY Gemfile.lock /home/rails_is_fun/Gemfile.lockRUN bundle install

COPY . /home/rails_is_funCMD bundle exec unicorn -p 3000

Even with this you have no concept of Gem cache, so any changes to Gemfile essentially means building them from scratch

Not to mention the burden of apt-get update/upgrade

What about secrets?When building application containers its pretty common to have secrets (API keys, DB urls etc.).

Dockerfile doesn't support any temporary/external files. While building you can't use VOLUMES either

No real solutions. Everything is bad (secrets in plaintext in layers, external wrappers etc)

#!/bin/bashgem install ejsonejson decrypt config/config.production.ejson > config.jsondocker build -t rails_is_fun .rm -rf config.json

The things we needed• Performance (not only for building a lot of containers but

building a single one quickly)

• Security. We handle money and peoples private information, so having a secure system is kinda important

• Coordinator for builds

Solution: Write your own builder

Our custom builder Locutus is a Golang app that:

• Listens to webhooks (every commit to GH triggers a build)

• Builds containers

• Pushes to our private registry

• Has a webgui showing the build step-by-step

• Sends hooks about build statuses

• Scales horizontally, eg. we can have N builders

The interesting part, buildingWe replaced Dockerfiles with prepare, precompile and compile scripts (vanilla Bash)

These scripts actually represent a different build phase and live in the app repo

Each phase does a docker run with the script and saves the container with docker commit.

prepareis used for your system packages (eg. apt-get install libmysqlclient-dev).

In this phase we only copy the script itself to our baseimage

We cache this image based on the md5sum of the prepareDuring Shopify builds we really rarely have to run this phase since system level depencies doesn't change often.

precompileis used for application dependencies and asset building (eg. bundle install)Used for creating a folder of wanted artifacts. Only thing that is persisted is /artifactsRun on every build, but build upon the previous cache of the specific app (and branch). So we never end up pulling all gems because of a Gemfile change etc.

compileis used for moving the generated arfifacts to correct folders (eg. .bundle) and finishing setup

Run on every build, but super fast since we are just doing bunch of mv and rm (to cleanup caches etc. to slim the image)

The result of this phase is the final image we actually can deploy to be run.

Build times around 40-50 seconds thanks to heavy caching

Containers never see the keys that are used for decrypting secrets

Questions?ps. http://www.shopify.com/careers