Post on 16-Apr-2017
Continuous Deployment with Kubernetes, Docker
and GitLab CI
@alexander_kielClojure Berlin 2016
Outline• Continuous Deployment why?
• Docker
• Kubernetes
• Sample Clojure Service
• Deploy with GitLabCI
Continuous Deployment• What do we want?
• Increase responsiveness
• Decrease time to market
• Gain confidence by deploying often in small amounts
• How to achieve that?
• Automate everything
• Always deploy the master into production
• Use feature toggles when needed
Simple Git Workflow
• Works for in-house apps
• not for libs or shipping apps
• No versions, no tags, just SHA’s
• Latest commit on master is always deployed to production
• Feature/fix branches are merged when ready
masterfeature/fixbranches
1ebb95d
be61dda
6e4010d
Docker• Like VM’s but much more light-weight and shippable
• Runs on Linux, executes processes in an isolated environment (resource limitation, filesystem, network)
• Container principle: Can contain everything, but looks the same from the outside
• A container platform can run every container
• Developers have max. freedom what to do
• In contrast: PaaS like Heroku - has to support the language
Kubernetes• Container runtime platform
• Originally designed by Google - now Open Source
• One of the most active projects on GitHub - 20,000 stars, 40,000 commits, 15,000 issues, 200 releases
• Alternatives: Apache Mesos, Docker Swarm (lacks features)
Kubernetes Architecturek8s-master-1
k8s-master-2
k8s-master-3
load-balancer-1
load-balancer-2
DNS RR
k8s-worker-1
proxy
app-1
k8s-worker-2
proxy
app-2
k8s-worker-n
proxy
app-k
etcd clusterquorum
HAProxy
• Runs on VMware ESX
• CoreOS Linux
• Single YAML file as configuration
• Everything in containers
Kubernetes - Pods• A Pod is a deployable unit in
Kubernetes
• Pods can contain multiple containers
• Containers inside a Pod share on port space, can use localhost and can communicate via IPC and shared memory
• Idea: one process per container - many cooperating processes in one Pod
apiVersion: v1 kind: Pod metadata: name: <pod-name> labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>
Kubernetes - Deployments
• A Deployment ensures that certain number of Pods are always running
• It consists of a Pod template and the number of replicas
• It supports hot-redeployments by changing parts of the Pod template
• Horizontal scaling is possible
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: <deployment-name> spec: replicas: 2 template: metadata: labels: <key>: <value> spec: containers: - name: <container-name> image: <container-image> ports: - containerPort: 80 env: - name: <key> value: <value>
Kubernetes - Services• Kubernetes uses an overlay
network to provide different address spaces (we use flannel)
• Every Pod has an IP address - but it changes every time one is created
• Services provide a stable IP address for groups of Pods
• Service names are resolvable by an internal DNS
• Service selectors are used to match Pods according to there labels
apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 80 protocol: TCP selector: service: clojure-berlin-2016
Kubernetes - External Access• Kubernetes networks are internal
only
• External access through load balancers necessary
• Certain Platforms like Google Compute Engine provide load balancer integration with Kubernetes
• We have our own solution as a combination of HAProxy and Kubernetes NodePort
• Kubernetes Services with type NodePort are exposed on every worker under a certain port
frontend http bind 0.0.0.0:80 mode http option httplog
acl host_clj hdr(host) clj.<domain>
use_backend clj if host_clj
backend clj mode http balance roundrobin option httplog server worker-1 <ip>:32599 check server worker-2 <ip>:32599 check
Deployment Lifecycle
GitLab CI
Source Code
build
test
Kubernetes Test
Cluster
Kubernetes Prod
Cluster
automatic deployment
manualdeployment
git push
Sample Clojure Service• .gitlab-ci.yml
• Like .travis.yml contains instructions for GitLabCI how to test, build and deploy
• Dockerfile
• Instructions for Docker how to build the image of the app
• Artifact of the build is a docker image - not uberjar
• kube-deployment.yml
• Kubernetes deployment instructions
• kube-svc.yml
• Kubernetes service description
https://github.com/alexanderkiel/clojure-berlin-2016
The Core Namespace
(ns clojure-berlin-2016.core (:require [aleph.http :as http] [clojure.core.async :refer [<!! chan]]))
(defn -main [& args] (-> (fn [_] {:status 200 :body "Clojure Berlin 2016"}) (http/start-server {:port 8080})) (<!! (chan)))
• A simple web server returning "Clojure Berlin 2016"
The Leiningen Project File
(defproject clojure-berlin-2016 "<VERSION>" :dependencies [[aleph "0.4.1"] [org.clojure/clojure "1.8.0"] [org.clojure/core.async "0.2.395"]]
:main clojure-berlin-2016.core)
• <VERSION> is replaced at build time by the Git SHA
• :main is for lein run to work
.gitlab-ci.yml - test/buildimage: clojure:lein-2.7.1
stages: - test - build - deploy
test: stage: test tags: - docker script: - lein test
build: stage: build tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" project.clj - docker build -t clojure-berlin-2016:$CI_BUILD_REF . - docker push clojure-berlin-2016:$CI_BUILD_REF
.gitlab-ci.yml - deploy branchdeploy-branch: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml except: - master when: manual
• Used to test a feature/fix branch in a full environment
.gitlab-ci.yml - deploy test
deploy-master: stage: deploy environment: test image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-test - kubectl apply -f kube-deployment.yml only: - master
.gitlab-ci.yml - deploy prod
deploy-prod: stage: deploy environment: prod image: dreg.life.uni-leipzig.local/kubectl:0.4 tags: - docker script: - sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml - kubectl config use-context gitlab-ci-prod-a - kubectl apply -f kube-deployment.yml only: - master when: manual
Docker fileFROM clojure:lein-2.7.1
COPY src /app/src COPY project.clj /app/
WORKDIR /app
RUN lein with-profile production deps
EXPOSE 80
CMD ["lein", "with-profile", "production", "run"]
• Just copy the sources into the container
• Use Leiningen itself to run in production
kube-deployment.ymlapiVersion: extensions/v1beta1 kind: Deployment metadata: name: clojure-berlin-2016 spec: replicas: 2 template: metadata: labels: app: lens service: clojure-berlin-2016 spec: containers: - name: clojure-berlin-2016 image: dreg.life.uni-leipzig.local/clojure-berlin-2016:<VERSION> ports: - containerPort: 8080 resources: requests: cpu: "125m" memory: "1Gi" limits: cpu: 1 memory: "2Gi"
kube-svc.yml
apiVersion: v1 kind: Service metadata: name: clojure-berlin-2016 labels: app: lens spec: type: NodePort ports: - port: 80 targetPort: 8080 protocol: TCP selector: service: clojure-berlin-2016
Steps to Follow• Create the Kubernetes Service
• kubectl create -f kube-svc.yml
• Edit HAProxy Config
• add rules and backend for the service
• Push to GitLab
• git push
Pipeline in GitLab CI
Deployment in GitLabCI
Environments in GitLabCI
• Very good visibility of wich commit is deployed in which environment right now
• Manual deployment to prod possible
Environment History
• Easy to see when what commit was deployed
• Rollback possible
Numbers• Our team has 4 developers
• We run 2 Kubernetes clusters (test and prod) with about 96 GB RAM and and 24 vCPU’s each
• We run about 60 pods in production
• We have other services like central log aggregation running using Fluentd and Elasticsearch/Kibana
Thank You• Sample Project on Github
https://github.com/alexanderkiel/clojure-berlin-2016
• Twitter@alexander_kiel
• Mailalexanderkiel@gmx.net