Scaling Open edX with KubernetesKubeCon London2016.03.10Morgan Robertson

Hi, I’m Morgan

What we’ll cover

● Introducing Open edX

● The problem: Walled gardens

● The solution: Open edX SaaS

● Open edX on Kubernetes

● Lessons learned

Introducing Open edX

Technical components

LMS/CMS (Django/Python)

Forum (Sinatra/Ruby)

User DB (MySQL)

Course DB (Mongo)

Tasks (Celery/RabbitMQ)

Caching (Memcache)

Proxy (Nginx)

Search (ElasticSearch)

Mapreduce (Hadoop)

The walled garden

Open edX Testdrive

Individual deployments don’t scale!

● Inefficient use of resources

● Can’t demo advanced features

● Painful upgrade process

Open edX SaaS

AMC

Kubernetes vs. Docker Swarm & friends

Kubernetes Swarm Compose

Scheduling ✔ ✔

Service discovery ✔ ✅ ✅

Container scaling ✔ ✔

Health checking ✔

Secret management ✔

Rolling updates ✔

Kubernetes vs. Docker Swarm & friends

Kubernetes Swarm Compose Consul

Scheduling ✔ ✔

Service discovery ✔ ✅ ✅ ✔

Container scaling ✔ ✔

Health checking ✔ ✔

Secret management ✔

Rolling updates ✔

Containerizing Open edX

edxapp

forum

Building images

FROM phusion/baseimage:0.9.15

# Clone the configuration repoRUN git clone https://github.com/appsembler/configuration.git \ /tmp/configuration

# Run AnsibleWORKDIR /tmp/configuration/playbooksRUN ansible-playbook -vvv -i 127.0.0.1, -c local forum.yml

CMD ["/sbin/my_init"]

Open edX on Kubernetes

Persistence: Mongo and MySQL

● Mongo replica set on VMs

● Google Cloud SQL

● Kubernetes services can abstract non-Pod backends

Using Endpoints for non-Pod backends# mongodb-endpoint.ymlapiVersion: v1kind: Endpointsmetadata: name: mongodbsubsets: - addresses: - ip: 10.240.0.3 - ip: 10.240.0.4 - ip: 10.240.0.5 ports: - name: mongodb port: 27017

# mongodb-service.ymlapiVersion: v1kind: Servicemetadata: name: mongodb labels: name: mongodbspec: clusterIP: None ports: - name: mongodb port: 27017

Initialization tasks using Jobs

# Create users and DBs in MySQLapiVersion: extensions/v1beta1kind: Jobmetadata: name: mysql-initspec: template: metadata: labels: name: mysql-init spec: containers: - name: mysql-init image: gcr.io/appsemblerx/mysql-init restartPolicy: OnFailure

● One-time tasks

● Batch tasks

● Run until success

Secrets

$ base64 secret.json

ewogICAgInNlY3JldCI6ICJteSBzZWNyZXQiCn0K

Kubernetes API

Pod 1secret. json

Pod 2secret. json

Google Cloud Monitoring

Lessons learned

● Containerizing an existing application can be difficult● Once containerized, deploying on Kubernetes is

generally straightforward● Not everything must be containerized to use

Kubernetes

Thank you for your time!