Docker in Production at the Aurora Team

29
Docker in production at the Aurora team Or: Now that we found docker what are we going to do with it? Timisoara Docker Meetup, Sept 4th 2017 Adina-Claudia Toma, Victor Dan Daneasa, Martin Danielsson

Transcript of Docker in Production at the Aurora Team

Page 1: Docker in Production at the Aurora Team

Docker in production at the Aurora team

Or: Now that we found docker what are we going to do with it?Timisoara Docker Meetup, Sept 4th 2017

Adina-Claudia Toma, Victor Dan Daneasa, Martin Danielsson

Page 2: Docker in Production at the Aurora Team

AgendaWho are we and why are we allowed to talk about this?

Our journey towards Docker in Production

What do you need for it?

How do you know it’s working?

Q & A

Page 3: Docker in Production at the Aurora Team

Who are we?

Page 4: Docker in Production at the Aurora Team

Aurora Project (iDesk2)Research Database for Lawyers,

Tax AccountantsLive with docker in Production

since MarchStill in transition towards

Microservices

Haufe Group~1600 employees in multiple countriesHQ in Freiburg, GermanyDevelopment Center in Timisoara

Adina-Claudia TomaSenior Developer

Victor Dan DaneasaSenior

Developer

Martin DanielssonSolution Architect

Who are we?

Page 5: Docker in Production at the Aurora Team
Page 6: Docker in Production at the Aurora Team

Moving from tinkering to production

Page 7: Docker in Production at the Aurora Team

Feels familiar?What’s this? $ docker pull postgres

$ docker run -d -p 5432:5432 postgres

AMAZEBALLS!I’m totally writing everything for docker!

A single VM will do. Right?

Dammit.

How hard can it be to get into production...

Page 8: Docker in Production at the Aurora Team

The Dev to Prod ChasmAmazeballs

factor

TraditionalOps Level Dev Tinkering Dev Process

SetupMature Production OpsIncl. CI/CD

Production Rollout

Page 9: Docker in Production at the Aurora Team

Building Blocks

Page 10: Docker in Production at the Aurora Team

● Hardware/Physical servers● OS● VM● Docker containers● Application

Levels of Abstraction

Page 11: Docker in Production at the Aurora Team

Building Blocks for running Docker in productionWhat you have to do by yourself:

● Image management● Container Orchestration● Automated CI/CD Pipelines● Log management● Monitoring on all levels● Data Persistence

What you can get for “free” if you use a cloud provider and orchestration framework:

● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure

Page 12: Docker in Production at the Aurora Team

Image Management● Consistent process to build and tag docker images● Private Docker image repository

○ Artifactory (JFrog)○ Azure Container Registry (ACR)○ Amazon EC2 Container Registry (ECR)○ Self-hosted with Docker○ Docker Hub○ Quay.io

● Security scanning of docker images for vulnerabilities

Page 13: Docker in Production at the Aurora Team

Container OrchestrationAbstracts the host infrastructure & allows to treat a cluster as a single deployment target

● Declarative configuration● Scheduling & high-availability● Service discovery & load-balancing● Health monitoring

Page 14: Docker in Production at the Aurora Team

Container Orchestration Tools

Page 15: Docker in Production at the Aurora Team

Providers

Page 16: Docker in Production at the Aurora Team

Automated CI/CD pipelines● Infrastructure as code● Reproducible environments● Automated tests● No downtime deployments:

○ Rolling updates○ Blue/green deployments○ Rollback on failure

Page 17: Docker in Production at the Aurora Team

Log Management

Page 18: Docker in Production at the Aurora Team

Data PersistenceContainers should be stateless.

State can be stored in:

● Data volumes per host -> non-portable between hosts● Shared filesystems: NFS, Ceph, GlusterFS● Docker volume plugins● Database/Storage as a service: AWS, Azure

Page 19: Docker in Production at the Aurora Team

our solution vectorWhat you have to do by yourself:

● Image management: private Haufe docker repository/Azure Container Registry● Container Orchestration: Kubernetes with Docker● Automated CI/CD Pipelines: Jenkins pipelines, bash, Ansible, Azure CLI● Log management: fluent-bit, fluentd, Graylog/Elasticsearch/Mongodb● Monitoring on all levels: Prometheus, Alertmanager, Grafana● Data Persistence: Postgres VM, NFS Server, Redis

What you can get for “free” if you use a cloud provider and orchestration framework:

● Azure Container Services Engine with Kubernetes● Security patches & restricted network access● Load balancing & service discovery● Automatic recovery from failure

Page 20: Docker in Production at the Aurora Team

Getting Application Insight (Or: How to not fly blind)

Page 21: Docker in Production at the Aurora Team

Prometheus● Whitebox monitoring● Scalable● Simple to setup● Discovery service● Built-in exporters (pull metrics)● Easy to integrate into your applications● PromQL (yet another query language)● Alerting included

Page 22: Docker in Production at the Aurora Team

Not fully blind And Getting better● Started with what we knew we need (the basics):

CPU, memory, IO

● Run into some problems:

Disk space, nodes failing, monitoring itself, API changes

● Things get better and better:

Alerting, app insights, moving parts

Page 23: Docker in Production at the Aurora Team

Overview

Page 24: Docker in Production at the Aurora Team

ApI Overview

Page 25: Docker in Production at the Aurora Team

Post MortemsResulted from a failure

Every member of the team participates

● What caused it?● What were the affected components?● Actions● Lessons learned

Page 26: Docker in Production at the Aurora Team

Tl;dr(Our) Conclusions

Page 27: Docker in Production at the Aurora Team

Trimmed for scale out

What we ended up with

Fully microservice enabled infrastructure

Insights on all levels

Full DevOps responsibility

Perhaps not what YOU need...Might single Docker Host be enough?

AWS Elastic Container Services?Docker DataCenter?k8s-as-a-service?

Traditional VMs?Google Container Engine?

Assess YOUR use case!

Page 28: Docker in Production at the Aurora Team

For us - absolutely worth the effort to

gain speed and flexibility

Invest only worth it with certain size and load

$Large upfront effort to get infrastructure

right

Your CI/CD pipelines are your safety net - make them rock solid

Practice provisioningdaily or weekly!

Steep learning curve If possible, start with something new, then move old workloads

Blue-eyed approach will fail - it is (a lot of)

work!

Many more moving parts - additional

complexity

Our conclusions and recommendations

Consider persistence early on

Page 29: Docker in Production at the Aurora Team

Q & A