Post on 15-Apr-2017
Clusternaut: Orchestrating Percona XtraDB Cluster
with Kubernetes.Raghavendra Prabhu
FOSDEM ’16rprabhu@yelp.com/me@rdprabhu.com/@randomsurfer
Yelp’s Mission:Connecting people with great
local businesses.
Yelp Stats:As of Q3 2015
89M 3271%90M
MeRaghavendra Prabhu
Software Engineer, Distributed Systems @ Yelp rprabhu@yelp.com / me@rdprabhu.com
Galera - “The Oar boat”
κυβερνήτης “The Helmsman”
DCOS
➔ Mesosphere ➔ Paasta➔ GCE
◆ Reference➔ ECS ➔ Smartcloud➔ Tectonic*
Rationale➔ Nodes v/s Hosts➔ Reusable components
◆ Monitoring◆ DNS◆ Logging◆ Metrics◆ Scheduler
➔ Agnostic ➔ Roles
The Fit➔ Layered
◆ Client - Server◆ Multi-layered
➔ Scaling◆ Horizontal and Vertical
● Preferred?➔ Statelessness in databases
◆ Planes of logic: Control/Data➔ Elasticity
◆ Elastic Scalability
Declarative vs Imperative➔ Configurable mgmt
◆ Puppet, Nix, Terraform ➔ Microservices
◆ What runs on my laptop ● What runs on server● Reproducibility
➔ 12-factor app➔ Composability ➔ Immutable deployment artifact
Containers ● Wth is a container and why should I care
○ Operating system virtualization● Isolation
○ Hierarchies of isolation - application, cgroups, namespaces, seccomp…
● Unikernels and VMs○ Role?
● Some - LXC, Docker*, Rocket*, runc, jails, solaris zones, lmcty, systemd-nspawn
Galera - really short intro! ➔ MySQL and WSREP api➔ Galera plugin➔ Group communication➔ Synchronous replication
◆ ‘Virtually’➔ EVS ➔ Certification-based
◆ Optimistic Concurrency➔ Automatic Node Provisioning
Galera - really short intro! ➔ CAP theorem and Galera
◆ CP➔ How does it fit
◆ Others◆ Idempotency
➔ Stateless? ◆ Symmetric◆ Replicas - Cassandra et.al.◆ MySQL Cluster
➔ Maintenance of Quorum
Orchestration● SOA def
○ ‘Stitching’○ ‘Composing’
● Automation?● Choreography● Best of both worlds
Kubernetes➔ Constituents:
◆ Kubelet◆ Pods
● Main service & helpers◆ Services
● The gcomm:// URL.◆ Replication Controller◆ Labels and Selectors
● MCollective
Kubernetes➔ Others:
◆ Mesos● Aurora, Chronos, Marathon
◆ Docker Swarm◆ Fleet
➔ Key Differences
Pods
Pods● Herd.. ● What should they contain - containers! ● How is the grouping done ● Pods and nodes
○ Colocation ● Pod communication● Labels
Services
Services● Don’t commingle with `microservices`● Think of endpoints. ● Layering architecture
○ Logical address of subset of pods● Communication
○ Environment■ Ordering requirement■ Discovery
○ DNS ■ Issues with DNS
Replication Controller● “Herd Management”● ASG● Pod template
○ Pattern○ Anti-pattern
● Role ○ Init for cluster○ Rolling updates ○ Multi-version
Networking● Docker-style linking● Proxy for Pods● Types
○ Pod to Pod○ Pod to Service○ Intra-Pod○ External to Service
● Providers:○ OpenVSwitch / Flannel / Calico / Weave / Google
External components● Flannel / Others● Etcd● Fluentd● Skydns● Container Registry● Scheduler● REST server● Proxy● cAdvisor
PAAS: PaaSTA● Docker● Mesos
○ Chronos○ Marathon
● Sensu● Smartstack
○ Zookeeper● Jenkins● Splunk / Signalfx● Why
Deployment● Declare and build individual Galera/PXC nodes.
○ Keep it minimal and simple○ No assumptions
● Without Kubernetes ○ Docker-compose
■ Possible issues ● Galera node ⇔ Pod
○ Haproxy○ xinetd
Deployment● Basic Steps:
○ Create a ‘flat’ network - 10.0.0.0/24○ Create a ‘cluster’ - zone○ Create a service endpoint.
■ Internal service - 3306/4567/4568.■ External service - 3306/3306(?).■ Expose the external.■ Session affinities.
Deployment● Next:
○ Bootstrap a node Pod from a template.■ Query existing with selector.
○ Start rest of nodes from template.■ Point to Service with selector.■ Replication controller
○ Volumes
Deployment - Implications● Load balancing in state transfers● Respawning of nodes on timeout
○ May not be same nodes.● kubectl to manage● Separation of client and cluster traffic
Service Definition
Dockerfile
```
... | kubectl create -f -
kubectl expose service eclient --port=3306 --target-port=3306 --name=loadbl --type='LoadBalancer’
```
``` .. | kubectl create -f -
kubectl scale --replicas=8 replicationcontrollers controller
```
```kubectl get --no-headers pods -l 'name=pxc' | wc -l2 kubectl stop …kubectl get --no-headers pods -l 'name=pxc' | wc -l2 ```
Credits! ● https://www.pinterest.com/duanejohnson851/star-trek-tng/● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_-
_dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png● https://camo.githubusercontent.
com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f672e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67
● http://galeracluster.com/documentation-webpages/_images/replicationapi.png
Further reading! ● http://kubernetes.io ● https://open.mesosphere.com/frameworks/ ● https://coreos.com/kubernetes/docs/latest/kubernetes-networking.
html● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html● http://12factor.net/ ● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html
ContactRaghavendra Prabhu
rprabhu@yelp.com / me@rdprabhu.com Twitter: @randomsurfer
Linkedin: rdprabhuGithub: ronin13
http://rdprabhu.com http://about.me/raghavendra.prabhu
http://rprabhu.yelp.com
@YelpEngineering
YelpEngineers
engineeringblog.yelp.com
github.com/yelp