Scaling Jenkins with Docker and Kubernetes

Scaling Jenkins with Docker and KubernetesCarlos Sanchez@csanchez

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterContainers & micro services

#jenkinsconf

#jenkinsconf

Footer

Scale

high availability distributed apps

Google starts 20billion/week

#jenkinsconf

Footer

But it is not trivial

#jenkinsconf

Footer


#jenkinsconf

Footer


#jenkinsconf

Footer

Service orientedthink about services, not machines

#jenkinsconf

Footer

Unix processes - lightweight VMs

#jenkinsconf

#jenkinsconf

FooterDockerLinux containersUnion File SystemFile SystemUsersProcessesNetwork

@YourTwitterHandle#jenkinsconf

#jenkinsconf

FooterDockerLinux requiredbutDocker Machine (formerly Boot2Docker) to the rescueOS XWindows

#jenkinsconf

#jenkinsconf

FooterDockerBuild once, run anywhere (kind of)Bare metalVirtual MachinesCloudDocker

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footerdeveloper orientedDependency hellinstallation nightmaresit ran on my machine

#jenkinsconf

#jenkinsconf

Footerops orientedno need to know internals of appsfocus on OPs problems(scale, monitoring,)clearer deliverables from dev

#jenkinsconf

#jenkinsconf

FooterDocker deliveryDocker imageDocker infrastructureAppSystem


#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterDockerdocker run -t -i ubuntu /bin/bashdocker ps -adocker build -t csanchez/test .

#jenkinsconf

#jenkinsconf

FooterRocketApp Container ImageApp Container RuntimeApp Container Discovery

#jenkinsconf

#jenkinsconf

FooterContainer Optimized Operating Systems

#jenkinsconf

#jenkinsconf

FooterContainer OSMinimal OSContainer optimizedNo package managerAutomatic atomic updatesEasy rollbacks

#jenkinsconf

#jenkinsconf

Footer

Kubernetes

#jenkinsconf

#jenkinsconf

FooterHow would you design your infrastructure if you couldn't login? Ever.

Kelsey HightowerCoreOS

#jenkinsconf

Footer

#jenkinsconf

FooterKubernetesContainer cluster orchestrationDocker containers across multiple hosts(nodes or minions)Higher level APIEnforced stateMonitoring of endpoints

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterMasterKubernetes API Serverscheduling and synchronizationetcdKubernetes Controller Manager Serverimplements replication algorithm watching etcd

#jenkinsconf

#jenkinsconf

FooterNodeDockerKubeletensures state of PodsKubernetes Proxysimple network proxyetcdSkyDNSElasticSearch + Kibana

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterProvidersGKEAzureVmwareRackspaceoVirtVagrantCloudStackUbuntu

#jenkinsconf

#jenkinsconf

FooterClusterexport KUBERNETES_PROVIDER=gceexport KUBERNETES_NUM_MINIONS=2cluster/kube-up.sh

#jenkinsconf

#jenkinsconf

Footer

Google Container Engine

#jenkinsconf

FooterGoogle Container Enginegcloud beta container --project my-project clusters create cluster-1 --machine-type g1-small --num-nodes 2

#jenkinsconf

#jenkinsconf

Footer

Tectonic by CoreOS

#jenkinsconf

#jenkinsconf

FooterNode--- kind: "Node" apiVersion: "v1" metadata: name: "127.0.0.1" selfLink: "/api/v1/nodes/127.0.0.1" uid: "8c3192d2-48ef-11e5-8d27-bae1092286ff" resourceVersion: "55118" creationTimestamp: "2015-08-22T17:02:19Z" labels: kubernetes.io/hostname: "127.0.0.1" spec: externalID: "127.0.0.1" status: capacity: cpu: "0" memory: "0" pods: "40" conditions: - type: "Ready" status: "True" lastHeartbeatTime: "2015-08-26T19:38:20Z" lastTransitionTime: "2015-08-26T12:18:48Z" reason: "kubelet is posting ready status" addresses: - type: "LegacyHostIP" address: "127.0.0.1" nodeInfo: ...

#jenkinsconf

FooterPodGroup of colocated containersSame network namespace/IPEnvironment variablesShared volumeshost mountedempty volumesGCE data disksAWS EBS volumes

#jenkinsconf

#jenkinsconf

FooterPodsNodeclientkubeletpodsMaster

NodekubeletpodsNodekubeletpods


#jenkinsconf

FooterPod--- kind: "Pod" apiVersion: "v1" metadata: name: "jenkins" labels: name: "jenkins" spec: containers: - name: "jenkins" image: "csanchez/jenkins-swarm:1.609.2" ports: - containerPort: 8080 hostPort: 8090 - containerPort: 50000 hostPort: 50000 volumeMounts: - name: "jenkins-data" mountPath: "/var/jenkins_home" volumes: - name: "jenkins-data" hostPath: path: "/home/docker/jenkins"

#jenkinsconf

FooterReplication controllerEnsure a number of pods are runningPod templatesRolling update

podpodpodcontainer 1container 2container 3

#jenkinsconf

#jenkinsconf

FooterReplication controllersNodeclientpodAMaster

NodepodANodepodB

podApodB


#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterReplication controller--- apiVersion: "v1" kind: "ReplicationController" metadata: name: "jenkins-slave" labels: name: "jenkins-slave" spec: replicas: 1 template: metadata: name: "jenkins-slave" labels: name: "jenkins-slave" spec: containers: - name: "jenkins-slave" image: "csanchez/jenkins-swarm-slave:2.0" env: command:

#jenkinsconf

#jenkinsconf

FooterReplication controllercommand: - "/usr/local/bin/jenkins-slave.sh" - "-master" - "http://$(JENKINS_SERVICE_HOST):$(JENKINS_SERVICE_PORT)" - "-tunnel" - "$(JENKINS_SLAVE_SERVICE_HOST):$(JENKINS_SLAVE_SERVICE_PORT)" - "-username" - "jenkins" - "-password" - "jenkins" - "-executors" - "1"

#jenkinsconf

#jenkinsconf

FooterServicesPod discoveryIP per serviceRoute to pods selected with labelsCan create a load balancer in GCE and AWS

#jenkinsconf

#jenkinsconf

FooterServicesclientpodService

podpod


#jenkinsconf

FooterServices--- apiVersion: "v1" kind: "Service" metadata: name: "jenkins" spec: type: "NodePort" selector: name: "jenkins" ports: - name: "http" port: 8090 nodePort: 32080 protocol: "TCP"

--- apiVersion: "v1" kind: "Service" metadata: name: "jenkins-slave" spec: type: "NodePort" selector: name: "jenkins" ports: - name: "http" port: 50000 nodePort: 32050 protocol: "TCP"

#jenkinsconf

FooterNetworkingall containers can communicate with all other containerswithout NATall nodes can communicate with all containers (and vice-versa) without NATthe IP that a container sees itself as is the same IPthat others see it as

Containers in a Pod can talk using localhost

#jenkinsconf

#jenkinsconf

FooterNetworkingEvery machine in the cluster is assigned a full subnetie. node A 10.0.1.0/24 and node B 10.0.2.0/24Simpler port mappingOnly supported by GCE

CoreOS flannelCreates an overlay network in other providers

#jenkinsconf

#jenkinsconf

FooterRelated projects

#jenkinsconf

#jenkinsconf

FooterDocker MachineProvision Docker enginesVirtualBox, replaces boot2docker !Amazon EC2Microsoft AzureGoogle Compute EngineOpenStackRackspaceVMware

#jenkinsconf

#jenkinsconf

FooterDocker SwarmClustering for Docker containersUsing the same APIIntegrates with Mesos / MesosphereAnd plannedAmazon EC2 Container Service (ECS)Google KubernetesIBM Bluemix Container ServiceJoyent Smart Data CenterMicrosoft Azure

#jenkinsconf

#jenkinsconf

FooterDocker ComposeOrchestration of multi-container appsBased on FigDefined by:containersconfigurationlinksvolumes

#jenkinsconf

#jenkinsconf

FooterApache MesosA distributed systems kernel

Docker ContainerizerMarathon & Chronos

#jenkinsconf

#jenkinsconf

FooterFlockerdata volume managermulti-host Docker cluster management toolZFS volumeshttps://github.com/ClusterHQ/flocker

#jenkinsconf

#jenkinsconf

Footer

Weave

#jenkinsconf

#jenkinsconf

FooterKubernetes and Jenkins

#jenkinsconf

#jenkinsconf

FooterKubernetes cluster with docker-compose# Docker Compose definition for a one node Kubernetes cluster# Based on Docker Cookbook example# https://github.com/how2dock/docbook/ch05/dockeretcd: image: kubernetes/etcd:2.0.5.1 net: "host" command: /usr/local/bin/etcd --addr=127.0.0.1:4001 --bind-addr=0.0.0.0:4001 --data-dir=/var/etcd/datamaster: image: gcr.io/google_containers/hyperkube:v1.0.1 net: "host" volumes: - /var/run/docker.sock:/var/run/docker.sock command: /hyperkube kubelet --api_servers=http://localhost:8080 --v=2 --address=0.0.0.0 --enable_server --hostname_override=127.0.0.1 --config=/etc/kubernetes/manifestsproxy: image: gcr.io/google_containers/hyperkube:v1.0.1 net: "host" privileged: true command: /hyperkube proxy --master=http://127.0.0.1:8080 --v=2 https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterJenkins master pod kind: "Pod" apiVersion: "v1" metadata: name: "jenkins" labels: name: "jenkins" spec: containers: - name: "jenkins" image: "csanchez/jenkins-swarm:1.609.2" ports: - containerPort: 8080 hostPort: 8090 - containerPort: 50000 hostPort: 50000https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

Footerstorage options volumeMounts: - name: "jenkins-data" mountPath: "/var/jenkins_home" volumes: - name: "jenkins-data" hostPath: path: "/home/docker/jenkins"# gcePersistentDisk:# pdName: my-data-disk# fsType: ext4# awsElasticBlockStore:# volumeID: aws:///# fsType: ext4https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterJenkins master services--- apiVersion: v1" kind: "Service" metadata: name: "jenkins" spec: type: "NodePort" selector: name: "jenkins" ports: - name: "http" port: 8090 nodePort: 32080 protocol: TCP"

--- apiVersion: "v1" kind: "Service" metadata: name: "jenkins" spec: type: "NodePort" selector: name: "jenkins" ports: - name: "http" port: 8090 nodePort: 32080 protocol: "TCP"https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterJenkins slaves replication pool--- apiVersion: "v1" kind: "ReplicationController" metadata: name: "jenkins-slave" labels: name: "jenkins-slave" spec: replicas: 1 template: metadata: name: "jenkins-slave" labels: name: "jenkins-slave" spec: containers: https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterJenkins slaves replication pool - name: "jenkins-slave" image: "csanchez/jenkins-swarm-slave:2.0" command: - "/usr/local/bin/jenkins-slave.sh" - "-master" - "http://$(JENKINS_SERVICE_HOST):$(JENKINS_SERVICE_PORT)" - "-tunnel" - "$(JENKINS_SLAVE_SERVICE_HOST):$(JENKINS_SLAVE_SERVICE_PORT)" - "-username" - "jenkins" - "-password" - "jenkins" - "-executors" - "1"https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterJenkins cluster in Kuberneteskubectl get nodeskubectl create --validate -f pod.ymlkubectl get podskubectl create --validate -f service-http.ymlkubectl create --validate -f service-slave.ymlkubectl get serviceskubectl create --validate -f replication.ymlkubectl get podskubectl scale replicationcontrollers --replicas=20 jenkins-slave

https://github.com/carlossg/kubernetes-jenkins

#jenkinsconf

#jenkinsconf

FooterKubernetes Jenkins plugin

#jenkinsconf

#jenkinsconf

FooterKubernetes Jenkins pluginAs a pluginon demand slaves

https://github.com/jenkinsci/kubernetes-plugin

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterKubernetes Jenkins pluginpods, not replication controllersJenkins Cloud APIFabric8 Java APIWorkflow support

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

Footer

#jenkinsconf

#jenkinsconf

FooterRoadmapStable APIWhen Kubernetes Java lib is stableUsing new Jenkins Cloud/Containers APIs

#jenkinsconf

#jenkinsconf

FooterExample code and slidesAvailable athttp://slideshare.csanchez.orghttps://github.com/carlossg/kubernetes-jenkins http://blog.csanchez.org

#jenkinsconf

#jenkinsconf

Footer

Thanks!

#jenkinsconf

FooterThanks to our Sponsors!

#jenkinsconf

Footer

Scaling Jenkins with Docker and Kubernetes

Engineering

Transcript of Scaling Jenkins with Docker and Kubernetes