Diving Through the Layers:Investigating runc, containerd and the Docker engine architecture

Phil EstesContainer Strategy LeadIBM Cloud Open TechnologiesTwitter: @estesp

Phil EstesSenior Technical Staff MemberIBM Cloud, Open Technologies

Container Strategy/Open Source Leader

Docker community core engine maintainer <Linux/open source expertise for 15 years @ IBM <

Community activities & accomplishments> Brought user namespace support to the Docker engine> Helped design v2.2 image specification with multi-platform support> Implemented first tool to create multi-platform images in Docker v2.3 registry> Founding maintainer of the new “containerd” project> Member of the “Docker Captains” program

Docker Engine History

• Single statically-linked binary contained all Docker engine functionality• Easy to install (no distro

dependencies)• Easy to upgrade• No separation of concerns;

privilege shared across all functions

• Daemon death = container death

2013/2014 - Early 2015

Docker Engine History• Little change to engine binary

architecture• Plugin support added• Many other ecosystem

pieces introduced

2H 2015 - Early 2016

The engine split!

• Introducing three new components to the engine architecture:• containerd: A lightweight daemon for

handling container lifecycle; gRPC API, simple `ctr` client for handling commands

• containerd-shim: A shim process for holding parent ownership (of a contained process) to allow daemon and containerd to exit/restart without impact

• runc: the OCI-compliant runtime for executing container processes given a filesystem bundle and OCI configuration

Docker 1.11 - April 2016

{

Why should you care?

> Improved (Docker) Engine Architecture

• Users of Docker engine have access to --live-restore (daemon restart without container restart)

• Pluggable OCI compliant runtimes (hyper.sh [runv], runz, etc)

> OCI compliance for Docker and other ecosystem participants

• Promise of better interoperability for runtimes and images• Cross-platform enablement managed at lower layer

> Lower-level Entrypoint(s) to Container Technology

• Highly performance sensitive container lifecycle requirements

• Docker architecture/feature set mismatch with use case

OCI & runc

> Announced June 20th, 2015 > Charter signed on December 8th, 2015 > 49 member companies > Both specifications nearing 1.0 release targets

https://opencontainers.orghttps://github.com/opencontainers

> runc is a client wrapper around libcontainer> Libcontainer is the OS level interface for containers

Other platforms and architectures can implement the libcontainer API via their own primitives/system-level container concepts

$ docker run -it --read-only \ -v /host:/hostpath \ alpine sh/#

{ "ociVersion": "1.0.0-rc3", "platform": { "os": "linux", "arch": "amd64" }, "process": { "terminal": true, "args": [ "sh" ], "env": [ "PATH=/usr/local/sbin:/usr/local/bin:/bin”

config.json

• A Linux Foundation Collaborative Project• Free from control by any particular vendor’s specific cloud stack or ecosystem• Includes a specification, reference runtime* and now, a specified image format

*seeded with runc + libcontainer by Docker

Using runc# runc --help

COMMANDS: checkpoint checkpoint a running container create create a container delete delete any resources held by one or more containers events display container events such as OOM notifications, cpu, memory, and IO usage statistics exec execute new process inside the container kill kill sends the specified signal (default: SIGTERM) to the container's init process list lists containers started by runc with the given root pause pause suspends all processes inside the container ps ps displays the processes running inside a container restore restore a container from a previous checkpoint resume resumes all processes that have been previously paused run create and run a container spec create a new specification file start executes the user defined process in a created container state output the state of a container update update container resource constraints

# runc run -b ~/containers/ubuntu ubunturoot@ubuntu:/# exit

#

runc needs two key pieces of information to create a container:

1. An OCI runtime configuration file (config.json)

2. A root filesystem (bundle) containing the container filesystem

> ocitools, riddler, or `runc spec` can create a simple starting point configuration for your container.

> tools for creating system images, including exporters for Docker or OCI images to a filesystem (see `skopeo` as well), can be used to create the filesystem

Live demo from DockerCon runc talk: https://www.youtube.com/watch?v=ZAhzoz2zJj8

Containerd announcement

> Will move to public foundation for governance in 1Q2017

> More base Docker engine functionality will move to containerd

> Will be used by the Docker engine when it reaches 1.0 functionality

Dec. 14, 2016: Docker announced the new* containerd project

* new in the sense of direction, governance, and content; containerd has existed since Docker 1.11

Using containerdNAME: ctr - High performance container daemon cli

USAGE: ctr [global options] command [command options] [arguments...] VERSION: 0.2.3 commit: 9c21822f344b7894ec2e55ac4b6d4e86118907dd COMMANDS: checkpoints list all checkpoints containers interact with running containers events receive events from the containerd daemon state get a raw dump of the containerd state version return the daemon version help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --debug enable debug output in the logs --address "unix:///run/containerd.sock" proto://address of GRPC API --conn-timeout "1s" GRPC connection timeout --help, -h show help --version, -v print the version

# ctr containers start {ID} {bundlePath}

> IMPORTANT: The expected method for interfacing and using containerd is via the gRPC API as an embedded container runtime.

> ctr is a simple (unofficial) client for debug, admin, and development interaction with the containerd daemon.

> containerd defaults to use runc as the OCI-compliant executor, but can be replaced by specifying a runtime and runtime arguments.

> Note that runc and containerd both support utilizing criu on Linux for checkpoint/restore of containers.

Watch this space as v0.2.3 moves towards v1.0.0 in 2017.

Docker engine: using containerd & runc

Containerd is used mostly as a orchestrator for the runc processes (supervisor, events, monitor signals/setup IO) which are executing containers.

Image handling/distribution and networking concerns continue to be part of the Docker engine, as well as higher layer features like Swarm, build, and OS integration capabilities like VPNKit, HyperKit, & DataKit.

TODAY::

Docker engine: using containerd & runc

Containerd 1.0 expands functionality to become a full-featured embeddable container runtime.

Higher layer concerns like Swarm, build, and OS integration capabilities via VPNKit, HyperKit, & DataKit remain as they are today, but the low-level engine features around layer storage, distribution system, and basic networking move into containerd.

FUTURE:

Use cases: Consuming the layers

> runc - no API, no daemon, no (combined) metadata/state• OCI compliance + high performance, low-level container executor• More flexible for configuration and filesystem generation

> containerd - gRPC API, daemon-based, events+supervisor• More focused, performance-oriented than entire Docker engine suite• Only container execution through OCI executor today, but 2017 will bring

significant additional capabilities.• Embed in broader orchestration or container-as-a-service offering:

• Kubernetes, Public Cloud (AWS, Azure, Bluemix), other orchestrators

> docker - JSON/REST API, daemon-based, full function lifecycle• Full-function but opinionated view of containers; available to client via

API/CLI, with remote API, Swarm built-in orchestration, SDN networking

Thank You!

@estesp

github.com/estesp

estesp@gmail.com

https://integratedcode.us

IRC: estesp