HELM Build fashionable container systems with Kubernetes · Author: Peter Rossbach Keywords:...

Copyright 2018 bee42 solutions gmbh <[email protected]> @PRossbach

rethink IT - We improve your systems with passion

1

HELM

Build fashionable container systems with Kubernetes



HELM

2

Originally developed by Deis 2015 and donated to Cloud Native Computing Foundation 2018

Second generation available and third generation development is started at July 2018

Package manager for Kubernetes clusters



What is Helm?

3

•Helm charts are build on top of Kubernetes manifests

•Charts are stored in a registry called Helm Museum

• It use a templating preprocessing and managed release of Kubernetes resources

•Ability to consider scalability from the get-go

•SRE’s and developer can search of charts and scratch it.



4

• Kubernetes is a container orchestrator.

• It’s how to run containers at scale.

• It’s a very active open-source platform with lots of

contributors, start at 6. June 2014

• Originally developed by Google and

donated to Cloud Native Computing Foundation



5



Kubernetes manifests

6



K8s Deployment Challenge

7

Move to microservice or serverless architecture. Applications consists from multiple components each component has its own resources and can be scaled individually.

It’s hard to ... ... manage, edit and update multiple K8s configurations ... deploy multiple K8s configurations as a SINGLE application ... share and reuse K8s configurations and applications ... parametrize and support multiple environments ... manage application releases: rollout, rollback, diff, history ... define deployment lifecycle (control operations to be run in different phases) ... validate release state after deployment



What Helm can do for you?

8

•Create new charts, aka package K8s manifests, from scratch

•Package charts into chart archive (tgz) file

•Interact with chart repositories where charts are stored

•Install and uninstall charts into an existing Kubernetes cluster

•Manage the release cycle of charts that have been installed with Helm



Chart

9

a package of Kubernetes manifests

Release a chart instance is loaded into Kubernetes

Repository a download area of published charts

Template a Kubernetes configuration file mixed with Go/Sprig templates

Concepts



How Helm 2 works?

10

client

api-service

tiller service(s)

https

grpc

brew install kubernetes-helm

curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash

or



helm init

11

$ cat >tiller-rbac.yaml <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system EOF $ kubectl create -f tiller-rbac.yaml $ helm init --service-account tiller --upgrade



Helm cli commands

12

completion Generate autocompletions script for the specified shell (bash or zsh) create create a new chart with the given name delete given a release name, delete the release from Kubernetes dependency manage a chart's dependencies fetch download a chart from a repository and (optionally) unpack it in local directory get download a named release history fetch release history home displays the location of HELM_HOME init initialize Helm on both client and server inspect inspect a chart install install a chart archive lint examines a chart for possible issues list list releases package package a chart directory into a chart archive plugin add, list, or remove Helm plugins repo add, list, remove, update, and index chart repositories reset uninstalls Tiller from a cluster rollback roll back a release to a previous revision search search for a keyword in charts serve start a local http web server status displays the status of the named release template locally render templates test test a release upgrade upgrade a release verify verify that a chart at the given path has been signed and is valid version print the client/server version information



Helm flow

13

client

api-service

tiller service(s)

publishfetch

installupgrade

delete

redis

create

application

tomcat

postgres



Example: Simple config map

14

$ helm create my-chart # review mychart templates $ rm -rf mychart/templates/*.* $ cat <<EOF >mychart/templates/configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: mychart-configmap data: myvalue: "Hello World" EOF $ helm install --dry-run --debug ./mychart $ helm install --name full-coral --namespace test ./mychart $ helm get manifest full-coral $ helm delete full-coral



Example: Prometheus & Grafana

15

# pre init # ————————————————————————————————————————————————————————————————————————— $ helm init --client-only $ helm repo add stable \ https://kubernetes-charts.storage.googleapis.com/ $ helm repo add incubator \ https://kubernetes-charts-incubator.storage.googleapis.com/ $ helm repo update

# install prometheus # ————————————————————————————————————————————————————————————————————————— $ helm install stable/prometheus \ --name prometheus \ --namespace monitoring \ --set rbac.create=true,server.persistentVolume.enabled=false,\ alertmanager.enabled=false,pushgateway.enabled=false

# install grafana # ————————————————————————————————————————————————————————————————————————— $ helm install ./grafana \ --name grafana \ --namespace monitoring \ --set server.persistentVolume.enabled=false,server.adminPassword=admin



Example: Traefik

16

$ helm install stable/traefik \ --name traefik --namespace kube-system \ --set rbac.enabled=true,serviceType=NodePort $ kubectl run —image=bee42/whoami:2.0.0 \ --expose --port=80 --target-port=80 --name=whoami $ cat >whoami-ingress.yaml <<EOF apiVersion: extensions/v1beta1 kind: Ingress metadata: name: whoami annotations: kubernetes.io/ingress.class=traefik spec: rules: - http: paths: - path: / backend: serviceName: whoami servicePort: http EOF $ kubectl apply -f whoami-ingress.yaml $ curl <node ip>:<traefik node.port>



Helm chart directory

17

<Helm chart> ├── Chart.yaml ├── <LICENSE>.md ├── <README>.md ├── charts │ ├── <external chart archive>.tgz │ ├── <subchart> │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ └── <k8s manifest>.yaml │ │ └── values.yaml ├── requirements.lock ├── requirements.yaml ├── templates │ ├── _helpers.tpl │ ├── NOTES.txt │ └── <k8s manifest>.yaml └── values.yaml



Chart.yaml

18

apiVersion: v1 description: A Helm chart for Docker Voting App porting to k8s cluster name: voting-app version: 0.3.2 keywords: - demo - kubernetes - helm - voting-app home: https://bee42.com/ icon: https://bee42.com/images/logo.svg sources: - https://gitlab.bee42.com/kubernetes/examples/voting-app maintainers: - name: Peter Rossbach email: [email protected]



values.yaml

19

# default ingress port servicePort: 80 # configure subchart redis redis: master: persistence: enabled: false usePassword: false … imageCredentials: registry: r.gitlab.bee42.com username: gitlab+deploy-token-1 password: xxx-xxx # ingress ingress: enabled: true nginx: enabled: true traefik: enabled: false



requirements.yaml

20

dependencies: - name: postgresql version: 0.13.1 repository: https://kubernetes-charts.storage.googleapis.com/ alias: postgresql-1 - name: redis version: 3.3.6 repository: https://kubernetes-charts.storage.googleapis.com/ condition: postgresql-1.enabled,global.result.enabled tags: - back-end - subchart2



_helper.tpl

21

{{/* vim: set filetype=mustache: */}} {{/* Expand the name of the chart. */}} {{- define "name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}}

{{/* Create a default fully qualified app name. We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). */}} {{- define "fullname" -}} {{- $name := default .Chart.Name .Values.nameOverride -}} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- end -}}

{{/* Create a registry image secret to pull voting app images */}} {{- define "imagePullSecret" }} {{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }} {{- end }}



Flow Control

22

If/Else

{{ if PIPELINE }} # Do something {{ else if OTHER PIPELINE }} # Do something else {{ else }} # Default case {{ end }}

data: myvalue: "Hello World" drink: {{ .Values.favorite.drink | default "tea" | quote }} food: {{ .Values.favorite.food | upper | quote }} {{- if eq .Values.favorite.drink "lemonade" }} mug: true {{- end }} # notice the "-" in the left, if will help eliminate newline before variable



Flow Control

23

With

data: myvalue: "Hello World" {{- with .Values.favorite }} drink: {{ .drink | default "tea" | quote }} food: {{ .food | upper | quote }} {{- end }} # instead of writing ".Values.favorite.drink"



Flow Control

24

Range

# predefined variable pizzaToppings: - mushrooms - cheese - peppers - onions

toppings: |- {{- range $i, $val := .Values.pizzaTopping }} - {{ . | title | quote }} # upper first character, then quote {{- end }}

sizes: |- {{- range tuple "small" "medium" "large" }} - {{ . }} {{- end }} # make a quick list



Flow Control

25

Variables

data: myvalue: "Hello World" {{- $relname := .Release.Name -}} {{- with .Values.favorite }} drink: {{ .drink | default "tea" | quote }} food: {{ .food | upper | quote }} release: {{ $relname }} {{- end }} # use variable in range toppings: |- {{- range $index, $topping := .Values.pizzaToppings }} {{ $index }}: {{ $topping }} {{- end }}

{{- range $key,$value := .Values.favorite }} {{ $key }}: {{ $value }} {{- end }} # instead of specify the key, we can actually loop through the values.yaml file and print values

values.yaml pizzaToppings: - mushrooms - cheese - peppers - onions



Flow Control

26

Nested Templates

# _helpers.tpl {{/* Generate basic labels */}} {{- define "my_labels" }} labels: generator: helm date: {{ now | htmlDate }} version: {{ .Chart.Version }} name: {{ .Chart.Name }} {{- end }}

# configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-configmap {{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template file. Without it version & name will not be generated. {{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to control indentation



Flow Control

27

Files inside Templates

# file located at parent folder # config1.toml: |- # message = config 1 here # config2.toml: |- # message = config 2 here # config3.toml: |- # message = config 3 here

data: {{- $file := .Files }} # set variable {{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list {{ . }}: |- # config file name {{ $file.Get . }} # get file's content {{- end }}



Flow ControlGlob-patterns & encoding

apiVersion: v1 kind: ConfigMap metadata: name: conf data: +{{ (.Files.Glob "foo/*").AsConfig | indent 2 }} --- apiVersion: v1 kind: Secret metadata: name: very-secret type: Opaque data: +{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }}

+token: |- + {{ .Files.Get "config1.toml" | b64enc }}

28



ImagePullSecrets

29

{{- define "imagePullSecret" }} {{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }} {{- end }}

imageCredentials: registry: quay.io username: someone password: sillyness

apiVersion: v1 kind: Secret metadata: name: imageSecretKey type: kubernetes.io/dockerconfigjson data: .dockerconfigjson: {{ template "imagePullSecret" . }}

values.yaml

templates/_helper.tpl

templates/imageSecretKey.yaml



Autodeploy

30

apiVersion: apps/v1 kind: Deployment spec: template: metadata: annotations: checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

deployment.yaml

or$ helm upgrade --recreate-pods



Helm hook flow

31

kubernetesmanifests

pre-install

post-install

pre-delete

post-delete

pre-upgrade

post-upgrade

pre-rollback

post-rollback

crd-install

apiVersion: ...kind: ...metadata: annotations: "helm.sh/hook": "pre-install"# ...



Helm: Hook reference

32

pre-install: Executes after templates are rendered, but before any resources are created in Kubernetes.

post-install: Executes after all resources are loaded into Kubernetes

pre-delete: Executes on a deletion request before any resources are deleted from Kubernetes.

post-delete: Executes on a deletion request after all of the release’s resources have been deleted.

pre-upgrade: Executes on an upgrade request after templates are rendered, but before any resources are loaded into Kubernetes (e.g. before a Kubernetes apply operation). post-upgrade: Executes on an upgrade after all resources have been upgraded.

pre-rollback: Executes on a rollback request after templates are rendered, but before any resources have been rolled back.

post-rollback: Executes on a rollback request after all resources have been modified. crd-install: Adds CRD resources before any other checks a



Helm hook flow

33

Chart load Chart verification

Pre-install hooksSorted by weight lowest first

Wait hooks ready

Load chart manifests

Post-install hookssorted by weight lowest first

Wait hooks ready

Return release nameHelm client exists



34



35

Text



Voting app

36

Web Ingress

Web Service Vote

Vote Pod Vote Pod

Redis Master Service

Redis Pod

Vote Result Pod Vote Result Pod

PostgreSQL Service

PostgreSQL Pod

Repl

icaS

et

Depl

oym

ent

Repl

icaS

et

Depl

oym

ent

Web Service Vote Result

Vote Worker Pod

Repl

icaS

et

Depl

oym

ent

Repl

icaS

et

Depl

oym

ent

Repl

icaS

et

Depl

oym

ent

DNS & LB

/<release-name>-vote /<release-name>-result

Redis Master Pod

Stat

eful

Set

PersistenceVolumeClaim

Redis Slave ServiceVolume (data)



Example: voting app

Dependencies

Subcharts

Templates

voting-app-reference ├── Chart.yaml ├── charts │ ├── postgresql-0.13.1.tgz │ ├── redis-3.3.6.tgz │ ├── result │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ ├── deployment.yaml │ │ │ └── service.yaml │ │ └── values.yaml │ ├── vote │ │ ├── Chart.yaml │ │ ├── templates │ │ │ ├── _helpers.tpl │ │ │ ├── deployment.yaml │ │ │ └── service.yaml │ │ └── values.yaml │ └── worker │ ├── Chart.yaml │ ├── templates │ │ ├── _helpers.tpl │ │ └── deployment.yaml │ └── values.yaml ├── requirements.lock ├── requirements.yaml ├── templates │ ├── _helpers.tpl │ ├── kcr-imageSecret.yaml │ ├── result-ingress.yaml │ └── vote-ingress.yaml └── values.yaml



Demo

38

• Review app code • Chart and Subchart review • Insides • Tips and Tricks

Work hard

Dream big



39

Text

http://voting-app-traefi-yujm3v.k8s.customer.bee42.cloud/dev-votehttp://voting-app-traefi-yujm3v.k8s.customer.bee42.cloud/dev-result



Kubernets stacking

40

Your multi tenant multi stage

multi branchesmulti services

SYSTEM

Kubernetes Platform

Application Services

k8s Extensions

Linux OS

vm or bare metal

k8s nodeCRI CNI CSI CLI

wireguard

operatorcontroller

alerts tracing

RBACSecurity

Authmetrics

dns lb/ingress

Repl

icaS

et

Depl

oym

ent

RBACServiceAccount

PodS

ecur

ityPo

licy

Net

Wor

kPol

icy

Lim

its

Auth

Podcontainer

container

Service

Ingress

Repl

icaS

etDe

ploy

men

t

RBACServiceAccount

PodS

ecur

ityPo

licy

Net

Wor

kPol

icy

Lim

its

Auth

Namespace Dev Namespace PreProd

Kube

rnet

s Ba

ckpl

ane

Podcontainer

container

Service

Ingress

loggingclusterAPI

Image repoBackup

Vault

SCMPipelines

Load Balancer / Router / DNS

Artefact repo

Serv

ice

Back

plan

e

Extensions

Volumes Network

Kubernetes extensions



Release a chart with security in mind

41



PodSecurityPolicy

42

apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: restricted annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: 'docker/default' apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default' seccomp.security.alpha.kubernetes.io/defaultProfileName: 'docker/default' apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default' spec: privileged: false # Required to prevent escalations to root. allowPrivilegeEscalation: false # This is redundant with non-root + disallow privilege escalation, # but we can provide it for defense in depth. requiredDropCapabilities: - ALL # Allow core volume types. volumes: - 'configMap' - 'emptyDir' - 'projected' - 'secret' - 'downwardAPI' # Assume that persistentVolumes set up by the cluster admin are safe to use. - 'persistentVolumeClaim' hostNetwork: false hostIPC: false hostPID: false runAsUser: # Require the container to run without root privileges. rule: 'MustRunAsNonRoot' seLinux: # This policy assumes the nodes are using AppArmor rather than SELinux. rule: 'RunAsAny' supplementalGroups: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535 fsGroup: rule: 'MustRunAs' ranges: # Forbid adding the root group. - min: 1 max: 65535 readOnlyRootFilesystem: false https://kubernetes.io/docs/concepts/policy/pod-security-policy/



Lessons learned

43

• Use the immutable deployment pattern

• Only system pods need access to the Api Server • Add PodSecurityPolicy and limits of the resource usages

• Limit the network access with a NetworkPolicy • Start with a Deny All Policy

• Use multiple tiller with limit RBAC roles and track your K8s cluster users • Check your container images with CVE Checkers (Clair, NeuVector, AquaSec, … )

• Add more Know How to your teams: Teach your Ops and Dev colleagues • Share your Knowledge and go to events, conferences and meetups

• Think about Chaos Engineering: Replace your system components every time and do that really! • Automate all what you can and never stop this doing!

• Don`t allow manuell manipulation at your K8s Cluster



44

Container System Blueprint: Beehive

Orchestration

Provisioning

Security

Config

Metrics

Alarming

Logging

Traceing

Artefact

Registry

Vault

Auth

Source Code

CI/CD Pipeline

Storage Network

API Gateway Routing

Loadbalancer, DNS and Firewall

On Premise Cloud

Operating Systems

Backends

VM machines

Databases

Bare metal

Container machinesContainer machines

Data

Bastion



CSSC: Container System Supply Chain

45



46



47

We hiring :-)

https://bit.ly/2K8DtRu [email protected]@bee42solutions



rethink IT

Build fashionable container systems with Kubernetes

48



bee42 Trainings

49

https://bee42.com/de/trainings



Kubernetes posterpre registration started

https://tinyurl.com/y9js3p7w

50

delivery starts today

PREVIEW

PREVIEW

WE



Cloud Native System Architect &

bee42 founder

Peter Roßbach@PRossbach

[email protected]

https://bee42.comhttps://devops-gathering.io

51

Save the date… #DOG19 11.-13. March 2019 at Bochum



References

52

[Sprig library](https://godoc.org/github.com/Masterminds/sprig[Deis Workflow](https://github.com/deis/workflow/tree/master/charts/workflow))[OpenStack chart](https://github.com/sapcc/openstack-helm)

[Helm.sh](https://helm.sh)

[Bitnami charts](https://github.com/bitnami/charts)[kubernetes charts](https://github.com/kubernetes/charts)

HELM Build fashionable container systems with Kubernetes · Author: Peter Rossbach Keywords:...

Documents

Transcript of HELM Build fashionable container systems with Kubernetes · Author: Peter Rossbach Keywords:...