The Challenges of Container Configuration

(without introducing more risk)

The Challenges of Container Configuration

PuppetGareth Rushgrove

New capabilities and associated problems


Gareth Rushgrove

@garethr


Gareth Rushgrove

(without introducing more risk)Configuration

What is it and why should I care


Gareth Rushgrove

(without introducing more risk)synonyms: design, grouping, marshalling

Gareth Rushgrove

(without introducing more risk)Marshalling your containers

Gareth Rushgrove


- Immutability and containers- Runtime vs build time- Who configures the orchestrator?

Gareth Rushgrove

(without introducing more risk)Mainly Docker and Kubernetes examples, but should be generally applicable

Gareth Rushgrove


Everything is immutable now?

Assumptions vs reality


Gareth Rushgrove


Gareth Rushgrove

$ docker run -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"


Gareth Rushgrove

$ docker exec a7a01beb14de touch /tmp/surprise


Gareth Rushgrove

$ docker diff a7a01beb14deC /tmpA /tmp/surprise

(without introducing more risk)Containers are notimmutable by default

Gareth Rushgrove

(without introducing more risk)Containers are not immutable by default

Gareth Rushgrove


Gareth Rushgrove

$ docker run --read-only -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"


Gareth Rushgrove

$ docker exec 379150b2cf05 touch /tmp/surprisetouch: cannot touch '/tmp/surprise': Read-only file system

(without introducing more risk)SuggestionEnable read-only where possible

Gareth Rushgrove

(without introducing more risk)Many applications won’t start with a read-only filesystem

Gareth Rushgrove


Gareth Rushgrove

1 import logging2 from logging.handlers import RotatingFileHandler3 4 from flask import Flask5 6 app = Flask(__name__)7 8 @app.route('/')9 def home():10 app.logger.info('log request')11 return "We'll never get to here"1213 if __name__ == '__main__':14 handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1)15 handler.setLevel(logging.INFO)16 app.logger.addHandler(handler)17 app.run(debug=True, host='0.0.0.0')


Gareth Rushgrove

$ docker run --read-only -p 5000:5000 garethr/flaskapp Traceback (most recent call last): File "app.py", line 14, in <module> handler = RotatingFileHandler('app.log', maxBytes=10000, backupCount=1) File "/usr/lib/python2.7/logging/handlers.py", line 117, in __init__ BaseRotatingHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/handlers.py", line 64, in __init__ logging.FileHandler.__init__(self, filename, mode, encoding, delay) File "/usr/lib/python2.7/logging/__init__.py", line 913, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 943, in _open stream = open(self.baseFilename, self.mode)IOError: [Errno 30] Read-only file system: '/app/app.log'

(without introducing more risk)tmpfs support added inDocker 1.10

Gareth Rushgrove


Gareth Rushgrove

$ docker run --read-only --tmpfs /tmp \ -d ubuntu:16.04 /bin/sh \ -c "while true; do echo hello world; sleep 1; done"


Gareth Rushgrove

$ docker exec 222331443d28 touch /tmp/surprise


Gareth Rushgrove

$ docker diff 222331443d28


Gareth Rushgrove

$ docker exec 222331443d28 ls /tmpsurprise

(without introducing more risk)SuggestionUse tmpfs only where needed

Gareth Rushgrove


RememberWithout technical controls you only have social guaranteesof immutability

Gareth Rushgrove

(without introducing more risk)Build vs Run

And the relationship between them


Given an image- What machine built this image?- Are all the licenses compatible?- Who supports this image?- Does this image contain malware?

Gareth Rushgrove


Given a running container- Who built it?- How was it built?- What software does it contain?- Is the software up-to-date?

Gareth Rushgrove


Gareth Rushgrove

FROM ubuntu:16.04

RUN apt-get update && \ apt-get install -y python-pip python-dev build-essential && \ apt-get clean && \ rm -rf /var/lib/apt/lists/*RUN pip install flask

COPY . /appWORKDIR /appENTRYPOINT ["python"]CMD ["app.py"]


Gareth Rushgrove

FROM ubuntu:16.04



Where did this base image come from?


Gareth Rushgrove

FROM ubuntu:16.04



What packages are installed? At what version?

Where are those packages from?


Gareth Rushgrove

FROM ubuntu:16.04



What version of flask is this?


Gareth Rushgrove

FROM ubuntu:16.04



What was in this folder at build time?

(without introducing more risk)The importance of time

Gareth Rushgrove

(without introducing more risk)How often are images rebuilt?

Gareth Rushgrove

(without introducing more risk)Rebuilding only on code change ignores environmental factors

Gareth Rushgrove

(without introducing more risk)Versioning and metadata

Gareth Rushgrove


Gareth Rushgrove

LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"


Gareth Rushgrove

LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"

What time? What timezone?


Gareth Rushgrove

$ docker inspect -f "{{json .Config.Labels }}" \ 4fa6e0f0c678 | jq{ "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12"}


SuggestionDecide upon and enforce metadata standards

Gareth Rushgrove


Gareth Rushgrove

LABEL com.example.git.repository="https://github.com/puppetlabs/puppet" \ com.example.git.sha="dc123cfb5ed4dca43a84be34a99d7c1080126fa4" \ com.example.build.time="2016-04-24T15:43:05+00:00" com.example.build.builder=“jenkins1.example.com" \ com.example.docs="https://github.com/puppetlabs/docker ...

(without introducing more risk)SuggestionEmbed Dockerfiles in images

Gareth Rushgrove


Gareth Rushgrove

$ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq{ "net.morethanseven.dockerfile": "/Dockerfile",}


Gareth Rushgrove

$ docker run -i -t garethr/alpine cat /DockerfileFROM alpineLABEL net.morethanseven.dockerfile="/Dockerfile"RUN apk add --update bash && rm -rf /var/cache/apk/*COPY Dockerfile /

(without introducing more risk)SuggestionProvide an API for your containers

Gareth Rushgrove


Gareth Rushgrove

$ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq{ "com.example.api.packages": "apk info -vv"}


Gareth Rushgrove

$ docker run -i -t garethr/alpine apk info -vvmusl-1.1.11-r2 - the musl c library (libc) implementationbusybox-1.23.2-r0 - Size optimized toolbox of many common UNIX utilitiesalpine-baselayout-2.3.2-r0 - Alpine base dir structure and init scriptsopenrc-0.15.1-r3 - OpenRC manages the services, startup and shutdown of a hostalpine-conf-3.2.1-r6 - Alpine configuration management scripts

(without introducing more risk)DEMO


Who configures the scheduler?

Higher level configuration


Schedulers/orchestrators abstract you from- Where individual containers run- Balancing due to new resources- Balancing due to failed resources

Gareth Rushgrove

(without introducing more risk)This results in a constraintsbased system

Gareth Rushgrove

(without introducing more risk)Which means those constraints need to be explicit and correct

Gareth Rushgrove


Gareth Rushgrove

$ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"


Gareth Rushgrove

$ docker run -d -P \ -e constraint:storage==ssd --name db mysql


Gareth Rushgrove

1 template:2 metadata:3 labels:4 app: guestbook5 tier: frontend6 spec:7 containers:8 - name: php-redis9 image: gcr.io/google_samples/gb-frontend:v410 resources:11 requests:12 cpu: 100m13 memory: 100Mi14 env:15 - name: GET_HOSTS_FROM16 value: dns17 # If your cluster config does not include a dns service, then to18 # instead access environment variables to find service host19 # info, comment out the 'value: dns' line above, and uncomment the20 # line below.

(without introducing more risk)How do you manage properties for all of your hosts?

Gareth Rushgrove


Gareth Rushgrove

$ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"

Does this machine really have an SSD?

What if someone swaps the drive?

(without introducing more risk)SuggestionUse properties of hosts as labels

Gareth Rushgrove


Gareth Rushgrove

$ facter | head -n 20 architecture => x86_64domain => localfacterversion => 2.4.6fqdn => Pro.localgid => staffhardwareisa => i386hardwaremodel => x86_64hostname => Proid => garethrinterfaces => lo0,gif0,stf0,en0,p2p0,awdl0,en1,en2,bridge0,vboxnet0,vboxnet1,vboxnet10,vboxnet11,vboxnet12,vboxnet13,vboxnet14,vboxnet2,vboxnet3,vboxnet4,vboxnet5,vboxnetipaddress => 192.168.0.5ipaddress_en0 => 192.168.0.5ipaddress_lo0 => 127.0.0.1


Gareth Rushgrove

$ facter -j os | jq { "os": { "name": "Darwin", "family": "Darwin", "release": { "major": "14", "minor": "5", "full": "14.5.0" } }}


Gareth Rushgrove

$ docker daemon \ --label net.example.os=`facter operatingsystem` \ --label net.example.virtual=`facter is_virtual` \ --label net.example.kernel=`facter kernelversion` \ ...


Orchestrators also tend to introduce new higher-level primitives

Gareth Rushgrove


Docker Networks, Kubernetes Services, ReplicationControllers, Chronos Jobs

Gareth Rushgrove

(without introducing more risk)Many with imperative interfaces

Gareth Rushgrove


Gareth Rushgrove

$ kubectl get pod mypod -o yaml \ | sed 's/$image: myimage$:.*$/\1:v4/' \ | kubectl replace -f -


Gareth Rushgrove

$ docker network create bobc0a0f4538d259515813b771264688d37aaedb41098379a0d73ec0ca08926fe68

$ docker network create bobError response from daemon: network with name bob already exists

And everything configuredin YAML

Gareth Rushgrove

Code plus data has advantagesover data alone

Gareth Rushgrove

The language to represent the data shouldbe a simple, data-only format such as JSON or YAML, and programmatic modification of this data should be done in a real programming language

Gareth Rushgrove

Borg, Omega, and Kubernetes, ACM Queue, Volume 14, issue 1 http://queue.acm.org/detail.cfm?id=2898444

“

Avoid repetitionCombine external inputsCorrectnessAbstractions

Gareth Rushgrove

----

SuggestionUse a higher level programming tool for generating config data

Gareth Rushgrove


Gareth Rushgrove

jsonnet.org


Gareth Rushgrove

$ cat test.jsonnet// Example jsonnet file{ person1: { name: "Alice", welcome: "Hello " + self.name + "!", }, person2: self.person1 { name: "Bob" },}


Gareth Rushgrove

$ jsonnet test.jsonnet | jq { "person1": { "name": "Alice", "welcome": "Hello Alice!" }, "person2": { "name": "Bob", "welcome": "Hello Bob!" }}


Gareth Rushgrove

$ jsonnet test.jsonnet | json2yaml --- person1: name: "Alice" welcome: "Hello Alice!" person2: name: "Bob" welcome: "Hello Bob!"


Gareth Rushgrove

garethr/kubernetes


Gareth Rushgrove

kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] },}


Gareth Rushgrove

apiVersion: v1kind: Podmetadata: namespace: default name: sample-podspec: container: - image: nginx name: container-name


Gareth Rushgrove

controller_service_pair { 'redis-master': app => 'redis', role => 'master', tier => 'backend', port => 6379,}


Gareth Rushgrove

apiVersion: v1kind: Servicemetadata: name: redis-master labels: app: redis tier: backend role: masterspec: ports: # the port that this service should serve on - port: 6379 targetPort: 6379 selector: app: redis tier: backend role: master---apiVersion: v1kind: ReplicationControllermetadata: name: redis-master # these labels can be applied automatically # from the labels in the pod template if not set labels: app: redis role: master tier: backendspec: # this replicas value is default # modify it according to your case replicas: 1 # selector can be applied automatically # from the labels in the pod template if not set # selector: # app: guestbook # role: master # tier: backend template:

(without introducing more risk)Conclusions

New technology means old problems


The difference between how you think something works and howit actually works riskshard-to-debug production issues

Gareth Rushgrove

(without introducing more risk)Containers introduce new and old configuration problems

Gareth Rushgrove


Configuration managementis the discipline aimed atminimising those risks

Gareth Rushgrove

(without introducing more risk)Start with principles

Gareth Rushgrove


- Identification- Control- Status accounting- Verification

Gareth Rushgrove

Military Handbook Configuration Management Guidance MIL-HDBK-61B


Apply them to yourcontainer basedinfrastructure today

Gareth Rushgrove

(without introducing more risk)Questions?And thanks for listening

The Challenges of Container Configuration

Technology

Transcript of The Challenges of Container Configuration