Live Container MigrationPhil Estes, STSM Open Containers, IBM Cloud, @estesp

Shaun Murakami, Senior Engineer, IBM Cloud, @stmuraka

Background• Live migration supported by most hypervisors in OpenStack

• Container use across OpenStack ecosystem growing

MAGNUM, MURANO, KOLLA, KURYR, KUBERNETES

• Why can’t I do live migration of containers?

http://docs.openstack.org/developer/nova/support-matrix.html

Why?• A hypervisor is controlling an emulated (complete) machine

• System suspend/resume is a solved problem• Hypervisor-based live migration can take advantage of this

• Containers are processes• Can the Linux kernel “migrate” a process?• Well, not as easily…

• Until CRIU• “Checkpoint Restore In Userspace”• Work began in 2011/2012 in the Linux kernel• 3.11 kernel + CRIU in late 2013 had working release• 2016: CRIU 2.0 released

CRIU: A Brief Overview• Proposed and led by Pavel Emelyanov from OpenVZ team

• Key difference from other checkpoint/restore attempts in Linux

was that the C/R work was performed from userspace

• All major kernel changes required were completed by the 3.11 Linux

kernel

• CRIU handles “freezing” all the process state:

• process/thread info+caps+uid/gid info, application memory, open files,

pipes/fifos, UNIX sockets, network sockets [magic!], IPC, timers, signals,

terminals, events/inotify state

See https://criu.org/Main_Page for more information.

CRIU & Docker• The Docker engine is bringing CRIU

support to the Docker API/client UI• Support has been merged into the

engine as of September 2016• Will be part of the Docker 1.13 release• Hard work (and perseverance) by Ross

Boucher

• Support already existed in libcontainer (and the runc) codebase• Popular demo from DockerCon SF 2015

migrating Quake game backend between hosts

Docker Implementation Overview

• Docker 1.13 use will require the experimental build to allow for testing/validation of capabilities and use cases

• Simplest model implemented allows for checkpoint associated with a specific container; restore re-starts same container at the named checkpoint

• Live migration not an intended feature. Don’t try this at home!

https://criu.org/Live_migration

# docker checkpoint [OPTIONS] container checkpoint_idOPTIONS: [--keep-running]

# docker start --checkpoint checkpoint_id [OPTIONS] container

Implementation Details

Two potential paths for live migration of container data:

memoryprocess state

filesystemcontents

memoryprocess state

filesystemcontents

1. Only in-memory changes to process state

2. Filesystem is untouched (e.g. a read-only root filesystem)

1

1. In-memory changes to process state

AND2. Changes to root

filesystem underneath container process

2

In-memory Only Sequence

• Checkpoint the container• Migrate checkpoint metadata to target• Pull (or pre-pull) exact image on target• Create container from same image at target• Extract checkpoint metadata into new container path• Start container with docker start --checkpoint

1

memoryprocess state

filesystemcontents

In-memory + Filesystem Sequence

• Checkpoint the container• Migrate checkpoint metadata to target• docker export stopped container, migrate to target

(docker import on target to re-create target image)• Read container metadata from source container• Create container from new image at target• Replicate container metadata into new container• Extract checkpoint metadata into new container path• Start container with docker start --checkpoint

2

memoryprocess state

filesystemcontents

Other Challenges

> Handling Container Metadata

• docker export -> docker import only addresses filesystem migration; does not save container runtime metadata (e.g. docker inspect information)

> Networking across Docker Hosts

• IP conflict with getting same IP assignment across hosts• Requires load balancer solution fronting migrating services

> Volume Support

• External volume data migration would require a solution that supported volume drivers/data export (use clustered FS)

Demo time!

Demonstrate basic checkpoint restore on a single host1

2

3

In-memory database migration across hosts

Video streaming container migration

Demo 1

A counter on the same host with checkpoint -> restore

busybox

HOST A

busybox’

HOST A

# docker run busybox1234<checkpoint>

# docker start --ch...<restore>5678910..

Demo 2

A tweet producer/consumer application centered around an in-memory database, redis.

redis

tweet2redis

load balancer(nginx)

HOST A HOST B

redis’tweetviewer(node.js)

HOST C

Demo 3

A video streaming back-end migration

VideoStream

load balancer(nginx)

HOST A HOST B

VideoStream’

HOST C

videoclient

Integration with OpenStack

• Magnum community integration of migration “steps”• Preconfigured nova instances with criu support/capabilities?

• Normalize networking aspect via Kuryr/overlay capabilities

• Use of SwarmKit/Docker with Swarm mode features for service

discovery (remove requirement on self-managed load balancer)

Thank You!