Kubernetes on Openstack - HPC Advisory Council
34
Transcript of Kubernetes on Openstack - HPC Advisory Council
© 2018 SWITCH | 2
Infrastructure & Data Services
Our offer
Our customers
Your added value
© 2018 SWITCH | 3
Your added value
SWITCH made – Swiss made
• Swiss law and data location• Scalable storage and computing power on
demand with immediate availability in accordance to the need of – and controlled by –the institutions
• Flexible usage and charging model, no up-front investment
• Simple administration; integrated into the academic network of SWITCH;security and identity services included
• Support for academic use cases • Created together with you
© 2018 SWITCH | 4
Our customers
Higher education• Cantonal universities• ETH domain with research institutions• Universities of applied sciences• Universities of teacher education
University-related organizations• Spin-Offs • Research institutions • eLearning Center • University hospitals
© 2018 SWITCH | 5
Our offer
• SWITCHengines• Virtual Private Cloud (VPC)• SCALE-UP (Project)*
* For developing academic services with 9 universities, as part of the „Scientific Information“ projects mandated by swissuniversities.
© 2018 SWITCH | 6
SWITCHengines
Customer tailored computing and storage performance for universities, research and teaching –further developed in the
SCALE-UP project mandated by swissuniversities.
Your benefits• Your data in Switzerland
• Integrated network and security
• Support for academic use cases
• Simple administration and billing
• Created together with you
Customers• Universities
• Research institutions
• eLearning Center
• University
hospitals
• Spin-Offs
Services• SWITCHengines (IaaS)
• Virtual Private Cloud (VPC)
• SCALE-UP (academic project)
© 2018 SWITCH | 7
Openstack Cloud Engineer at SWITCH. He works in NRENs since 2011, first in Italy and then in Switzerland. He has significant experience in running critical infrastructures using Open Source software. He works together with the Infrastructure & Data team, to deliver an Openstack based cloud to the Swiss Universities.
Saverio Proto
© 2018 SWITCH | 8
Container orchestration.Open Source system to deploy containerized applications.
• What you should already know:• Docker, build docker containers• Run your application in a container
• What Kubernentes will do for me ?• Manage the lifecycle of containers• Schedule the containers to the hosts• Attach storage volumes to the containers• Setup networking
Kubernetes (K8s): what is it ?
© 2018 SWITCH | 9
• K8s can interact with the Openstack API
• Use keystone for authentication• Create volumes• Create load balancers• Set routes in Neutron routers
Kubernetes on Openstack
© 2018 SWITCH | 10
• https://github.com/switch-ch/k8s-on-openstack/• Forked from https://github.com/infraly/k8s-on-openstack
This ansible playbook will create Openstack instances, and will install Kubernetes on them.
export KEY=keynameexport IMAGE="Ubuntu Xenial 16.04 (SWITCHengines)"export NETWORK=k8s( read REAME for complete variables list)
ansible-playbook site.yaml
Deploy K8s on Openstack
© 2018 SWITCH | 11
ubuntu@k8s-master:~$ kubectl get nodesNAME STATUS AGE VERSIONk8s-1 Ready 3m v1.9.4k8s-2 Ready 3m v1.9.4k8s-3 Ready 3m v1.9.4k8s-master Ready 4m v1.9.4
Start !
© 2018 SWITCH | 12
The playbook returns an admin.conf that looks like this:apiVersion: v1
clusters:
- cluster:
certificate-authority-data: [..CUT..]
server: https://<ipaddress>:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubernetes-admin
name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
user:
client-certificate-data: [..CUT..]
client-key-data: [..CUT..]
Client config
© 2018 SWITCH | 13
• kubectl is the command line tool• ~/.kube/config is the config file
• Supports Openstack Keystone authentication• kubectl is able to read usual Openstack OS_ env vars
• kubectl config set-credentials \
osuser --auth-provider=openstack
• source ~/openrc
Client config
© 2018 SWITCH | 14
Create a context to use the Openstack User
kubectl config set-context \
--cluster=kubernetes \
--user=osuser osuser@kubernetes
Lets use the contextkubectl config use-context osuser@kubernetes
Client config
© 2018 SWITCH | 15
Keystone authentication for k8s
Kubectlclient
OpenstackKeystone
KubeApiserver
WebHook
2. Requestwith bearer token
1. Token issue
3. Calling WebHook for token validation
4. Token validation with keystone
© 2018 SWITCH | 16
kube-apiserver--authentication-token-webhook-config-file=webhook.kubeconfig.yaml
---
apiVersion: v1
clusters:
- cluster:
insecure-skip-tls-verify: true
server: https://localhost:8443/webhook
name: webhook
contexts:
- context:
cluster: webhook
user: webhook
name: webhook
current-context: webhook
kind: Config
preferences: {}
users:
- name: webhook
Configure the webhook
© 2018 SWITCH | 17
It is just a container:https://hub.docker.com/r/zioproto/k8s-keystone-auth/
More recent source code can be found here:https://github.com/kubernetes/cloud-provider-openstack
make image-k8s-keystone-auth
Start the webhook container
© 2018 SWITCH | 18
• Keystone tested for Authentication
• Default RBAC for Kubernetes implements Authorization
• Keystone projects are mapped to Kubernetes Groups
Authorization via RBAC
© 2018 SWITCH | 19
/etc/kubernetes/cloud-config[Global]auth-url = https://keystone.cloud.switch.ch:5000/v2.0username = [email protected] = ******tenant-name = [email protected] = ZH
[BlockStorage]trust-device-path = falsebs-version = v2
[Route]router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d
[LoadBalancer]lb-version = v2floating-network-id = 3cc83f7d-9119-475b-ba17-f3510c7902e8subnet-id = 5e18c72c-1902-4846-bd84-ec54cf028375
K8s Master interaction with Openstack
© 2018 SWITCH | 20
/etc/kubernetes/manifests/kube-controller-manager.yaml
spec:
containers:
- command:
- kube-controller-manager
- --cloud-provider=openstack
- --cloud-config=/etc/kubernetes/cloud-config
Interaction with Openstack
© 2018 SWITCH | 21
I changed my the way of doing Networking after the Openstack summit in Sydney
Use --network-plugin=kubenet
• https://github.com/zioproto/k8s-on-openstack/commit/f4506ed202ecc6fc4ff5ac603fd28f3664cb2871
• https://www.openstack.org/videos/sydney-2017/kubernetes-on-openstack-the-technical-details
Kubernetes Neutron Networking
© 2018 SWITCH | 22
Kubernetes Neutron Networking
neutronrouter
VMaka k8s node
VMaka k8s node
Default gateway is the neutron router
Has static routes to reach Pod Networks
Default gateway is the neutron router
Pod Pod
Pod network assigned by the k8s master
Pod Pod
Pod network assigned by the k8s master
© 2018 SWITCH | 23
• Docker containers run into Openstack instances• Persistent storage is cinder volumes
• K8s will attach the Cinder Volume to the right Openstackinstance were the docker container is scheduled
volumes:
- name: mysql-persistent-storage
cinder:
volumeID: <uuid>
fsType: ext4
Persistent Storage
© 2018 SWITCH | 24
• This method is not practical• we have to manage Cinder volumes on the side• Export UUID
• Solution• Define a Storage Class
Persistent storage
© 2018 SWITCH | 25
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: defaultannotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/cinder
parameters:
type: fast
availability: nova
Storage Class
© 2018 SWITCH | 26
kind: PersistentVolumeClaimapiVersion: v1metadata:
name: mysql-pvcannotations:
volume.beta.kubernetes.io/storage-class: defaultspec:
accessModes:- ReadWriteOnce
resources:requests:
storage: 30Gi
Persistent Storage pvc
© 2018 SWITCH | 27
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pvc
Persistent Storage with pvc
© 2018 SWITCH | 28
I suggest not to use it, at least in Openstack Newton
• Both DNAT and SNAT are necessary• you lose the client IP information
• IPv6 support status is unknown to me
• It is a pure L4 loadbalancer, cannot terminate TLS connections or rewrite HTTP headers
Openstack LBaaSv2 with K8s
© 2018 SWITCH | 29
Openstack LBaaSv2
neutronrouter
VMaka k8s node
VMaka k8s node
Default gateway is the neutron router
Default gateway is the neutron router
Pod Pod
Pod network assigned by the k8s master
Pod Pod
Pod network assigned by the k8s master
neutronLBaaS
Svc FloatingIP configured here
Openstack project network
Both DNAT and SNAT are necessary here.
© 2018 SWITCH | 30
• An API object that manages external access to the services in a cluster, typically HTTP.
• Ingress can provide load balancing, TLS termination and name-based virtual hosting.
• https://kubernetes.io/docs/concepts/services-networking/ingress/
• At least two known implementations• Nginx based• HAProxy
Kubernetes NGINX Ingress
© 2018 SWITCH | 31
Kubernetes NGINX Ingress
VM
aka k8s node
VM
aka k8s node
Pod Pod
Pod network assigned
by the k8s master
Pod Pod
Pod network assigned
by the k8s master
VM k8s
master
Kubernetes-nginx-ingress
Default network namespace
(docker –net=host)
• It is a special privileged Docker container running NGINX
• On the k8s master (s) or dedicated VM
• Has access to external IP addresses and cluster IP addresses
Kubernetes cluster network
© 2018 SWITCH | 32
How I use the Ingress ?
© 2018 SWITCH | 33
Slack channel #sig-openstack• Join via http://slack.k8s.io/
Follow our cloud blog:• https://cloudblog.switch.ch/
Get in touch with me• [email protected]
Getting involved
© 2018 SWITCH | 34
Thank you ! Questions ?
