Docker in production at the Aurora team

Or: Now that we found docker what are we going to do with it?Timisoara Docker Meetup, Sept 4th 2017

Adina-Claudia Toma, Victor Dan Daneasa, Martin Danielsson

AgendaWho are we and why are we allowed to talk about this?

Our journey towards Docker in Production

What do you need for it?

How do you know it’s working?

Q & A

Who are we?

Aurora Project (iDesk2)Research Database for Lawyers,

Tax AccountantsLive with docker in Production

since MarchStill in transition towards

Microservices

Haufe Group~1600 employees in multiple countriesHQ in Freiburg, GermanyDevelopment Center in Timisoara

Adina-Claudia TomaSenior Developer

Victor Dan DaneasaSenior

Developer

Martin DanielssonSolution Architect

Who are we?

Moving from tinkering to production

Feels familiar?What’s this? $ docker pull postgres

$ docker run -d -p 5432:5432 postgres

AMAZEBALLS!I’m totally writing everything for docker!

A single VM will do. Right?

Dammit.

How hard can it be to get into production...

The Dev to Prod ChasmAmazeballs

factor

TraditionalOps Level Dev Tinkering Dev Process

SetupMature Production OpsIncl. CI/CD

Production Rollout

Building Blocks

● Hardware/Physical servers● OS● VM● Docker containers● Application

Levels of Abstraction

Building Blocks for running Docker in productionWhat you have to do by yourself:

● Image management● Container Orchestration● Automated CI/CD Pipelines● Log management● Monitoring on all levels● Data Persistence

What you can get for “free” if you use a cloud provider and orchestration framework:

● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure

Image Management● Consistent process to build and tag docker images● Private Docker image repository

○ Artifactory (JFrog)○ Azure Container Registry (ACR)○ Amazon EC2 Container Registry (ECR)○ Self-hosted with Docker○ Docker Hub○ Quay.io

● Security scanning of docker images for vulnerabilities

Container OrchestrationAbstracts the host infrastructure & allows to treat a cluster as a single deployment target

● Declarative configuration● Scheduling & high-availability● Service discovery & load-balancing● Health monitoring

Container Orchestration Tools

Providers

Automated CI/CD pipelines● Infrastructure as code● Reproducible environments● Automated tests● No downtime deployments:

○ Rolling updates○ Blue/green deployments○ Rollback on failure

Log Management

Data PersistenceContainers should be stateless.

State can be stored in:

● Data volumes per host -> non-portable between hosts● Shared filesystems: NFS, Ceph, GlusterFS● Docker volume plugins● Database/Storage as a service: AWS, Azure

our solution vectorWhat you have to do by yourself:

● Image management: private Haufe docker repository/Azure Container Registry● Container Orchestration: Kubernetes with Docker● Automated CI/CD Pipelines: Jenkins pipelines, bash, Ansible, Azure CLI● Log management: fluent-bit, fluentd, Graylog/Elasticsearch/Mongodb● Monitoring on all levels: Prometheus, Alertmanager, Grafana● Data Persistence: Postgres VM, NFS Server, Redis

What you can get for “free” if you use a cloud provider and orchestration framework:

● Azure Container Services Engine with Kubernetes● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure

Getting Application Insight (Or: How to not fly blind)

Prometheus● Whitebox monitoring● Scalable● Simple to setup● Discovery service● Built-in exporters (pull metrics)● Easy to integrate into your applications● PromQL (yet another query language)● Alerting included

Not fully blind And Getting better● Started with what we knew we need (the basics):

CPU, memory, IO

● Run into some problems:

Disk space, nodes failing, monitoring itself, API changes

● Things get better and better:

Alerting, app insights, moving parts

Overview

ApI Overview

Post MortemsResulted from a failure

Every member of the team participates

● What caused it?● What were the affected components?● Actions● Lessons learned

Tl;dr(Our) Conclusions

Trimmed for scale out

What we ended up with

Fully microservice enabled infrastructure

Insights on all levels

Full DevOps responsibility

Perhaps not what YOU need...Might single Docker Host be enough?

AWS Elastic Container Services?Docker DataCenter?k8s-as-a-service?

Traditional VMs?Google Container Engine?

Assess YOUR use case!

For us - absolutely worth the effort to

gain speed and flexibility

Invest only worth it with certain size and load

$Large upfront effort to get infrastructure

right

Your CI/CD pipelines are your safety net - make them rock solid

Practice provisioningdaily or weekly!

Steep learning curve If possible, start with something new, then move old workloads

Blue-eyed approach will fail - it is (a lot of)

work!

Many more moving parts - additional

complexity

Our conclusions and recommendations

Consider persistence early on

Q & A