Kubernetes on AWS

AT EUROPE’S LEADING

ONLINE FASHION PLATFORM

HENNING JACOBS

@try_except_

2017-03-27

2

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

ZALANDO

15 markets

6 fulfillment centers

20 million active customers

3.6 billion € net sales 2016

165 million visits per month

12,000 employees in Europe

3

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

ZALANDO TECHNOLOGY

HOME-BREWED,CUTTING-EDGE& SCALABLEtechnology solutions

>1,600employees from

tech locations+ HQs in Berlin6

77nations

help our brand toWIN ONLINE

4

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

KUBERNETES ON AWS: CONTEXT

200 engineering teams

30 prod. clusters

AWS

Dockerized apps

No manual operations

Reliability

Autoscaling

Seamless migration

5

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

ARCHITECTURE

6

Please write the title in all capital letters

ISOLATED AWS ACCOUNTS

Internet

*.abc.example.org *.xyz.example.org

Product ABC Product XYZ

EC2

LBLB

7

Please write the title in all capital letters

KUBERNETES ON AWS

8

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

ARCHITECTURE DECISIONS

• API server behind SSL ELB• Webhook for authn & authz

• OAuth Bearer token• Group membership lookup

• Read only access to production• CI/CD for write access• etcd running separately on EC2• Multi AZ clusters

9

Please write the title in all capital letters

CLUSTER PROVISIONING

10

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

CLUSTER PROVISIONING

• Two Cloud Formation stacks

• Master & worker ASGs + etcd

• Nodes w/ Container Linux

• K8s manifests applied separately

• kube-system Deployments

• DaemonSets

11

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

DEPLOYMENT

12

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

DEPLOYMENT CONFIGURATION

.├── apply│ ├── credentials.yaml # K8s TPR│ ├── ingress.yaml # K8s Ingress│ ├── redis-deployment.yaml # K8s Deployment│ ├── redis-service.yaml # K8s Service│ └── service.yaml # K8s Service├── deployment.yaml # K8s Deployment└── pipeline.yaml # proprietary config

13

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

JENKINS DEPLOY PIPELINE

14

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

INGRESS

15

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

INGRESS.YAML

apiVersion: extensions/v1beta1kind: Ingressmetadata: name: "{{ application }}" annotations: # optional: SSL certificate ARN to use for the ALB (auto discovery for ACM) zalando.org/aws-load-balancer-ssl-cert: "arn:aws:iam:..:..:..1a"spec: rules: # DNS name your application should be exposed on - host: "myapp.foo.example.org" http: paths: - backend: serviceName: "{{ application }}" servicePort: 80

16

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

INGRESS CONTROLLER

17

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

AWS INTEGRATION

18

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

CLOUD FORMATION VIA CI/CD

.├── apply│ ├── cf-iam-role.yaml # AWS IAM Role│ ├── cf-rds.yaml # AWS RDS Database│ ├── kube-ingress.yaml # K8s Ingress│ ├── kube-secret.yaml # K8s Secret│ └── kube-service.yaml # K8s Service├── deployment.yaml # K8s Deployment└── pipeline.yaml # CI/CD config

19

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

ASSIGNING AWS IAM ROLE TO POD

kind: Deploymentspec: template: metadata: annotations: # annotation for kube2iam iam.amazonaws.com/role: "app-{{ application }}-1" spec: containers: - name: ... ...

20

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

CLUSTERAUTOSCALING

21

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

CLUSTER AUTOSCALING

Control # of worker nodes in ASG:

• Satisfy all resource requests

• One spare node per AZ

• No manual config “tweaking”

• Scale down, but not too fast

22

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

CURRENT SETUP

• https://github.com/hjacobs/kube-aws-autoscaler

• Node draining via systemd unit

Open topic: node “readiness” during scale out

24

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

OAUTH / IAMINTEGRATION

25

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

DECLARING NEEDED CREDENTIALS

# apply/credentials.yamlapiVersion: "zalando.org/v1"kind: PlatformCredentialsSetmetadata: name: "{{ application }}"spec: application: "{{ application }}" tokens: # OAuth service tokens mytok: # the token name used in application code privileges: - com.zalando::foobar.write clients: # OAuth clients implicit: grant: implicit # grant type according to RFC-6749 realm: users redirectUri: https://myapp.foo.example.org/oauth

26

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

MOUNTING THE OAUTH CREDENTIALS

kind: Deploymentspec: template: spec: containers: - name: ... ... volumeMounts: - name: "{{ application }}-credentials" mountPath: /meta/credentials readOnly: true volumes: - name: "{{ application }}-credentials" secret: secretName: "{{ application }}"

27

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

USING THE OAUTH CREDENTIALS

#!/bin/bash

type=$(cat /meta/credentials/read-only-token-type)

secret=$(cat /meta/credentials/read-only-token-secret)

curl -H "Authorization: $type $secret" \

https://resource-server.example.org/protected

28

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

OPERATIONS&

MONITORING

29

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

OPERATIONS

• Cluster updates automatic via CLM

• CronJob is great, but needs cleanup

• Docker can be PITA

30

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

CLUSTER UPDATES

31

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

LIMIT RANGE

kubectl describe limitrange

Name: limits

Namespace: default

Type Resource Min Max Default Req Default Limit Max Limit/Request Ratio

---- -------- --- --- ----------- ------------- -----------------------

Container memory - 64Gi 100Mi 1Gi -

Container cpu - 16 100m 3 -

32

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

MONITORING

•

33

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

SIMPLE ZMON CHECK/ALERT EXAMPLE

•

34

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

MONITORING

• Each cluster contains ZMON appliance

• K8s resources are available as ZMON entities

• Users can create app checks/alerts via UI

https://github.com/hjacobs/kube-ops-view

36

Put images in the grey dotted box "unsupported placeholder" - behind the orange box (left side stays white)

Write the quote in all capital letters

OPEN SOURCE

37

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"

Use bullet points to summarize information rather than writing long paragraphs in the text box

OPEN SOURCE

Kube AWS Ingress Controllerhttps://github.com/zalando-incubator/kube-ingress-aws-controller

External DNShttps://github.com/kubernetes-incubator/external-dns

Zalando Cluster Config & Docshttps://github.com/zalando-incubator/kubernetes-on-aws

more to come...

Please write contact name, department and position in all capital letters

QUESTIONS?

HENNING JACOBS

TECH INFRASTRUCTURE

CLOUD ENGINEER

henning@zalando.de

@try_except_

Please write contact name, department and position in all capital letters