container the universal container runtimeJustin Cormack, Docker@justincormack

containerd: now a CNCF project

“containerd is a widely used container runtime with an emphasis on simplicity, robustness and portability”

“containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users. containerd will serve as a core container runtime for the CNCF ecosystem.”

2

containerd: cloud native

Uses

Can be used in

3

containerd: not a new project, an evolution

• Started in November 2015 to control runc, the OCI runtime• Used by Docker since 1.11 in 2016 as a container runtime• Relaunched in December 2016 with new scope• Docker now using 0.2 branch• 1.0 master branch is where the new work is taking place

entirely new scope, and donated to CNCF

4

containerd: What is a Core Container Runtime?

Component that provides core primitives to manage containers on a host

• Container execution and supervision• Image distribution• Network Interfaces and management• Local storage• Native plumbing level API

5

containerd: isn’t that what Docker does?

Docker provides much more than that, it is a platform

• user interface• backward compatibility with existing applications• clustering with Swarm mode• opinionated workflows and defaults, such as Docker Hub• commercial support• product direction not entirely community led

Some people do not want those things, containerd is for them.

6

containerd: role in Container Ecosystem

7

containerd and Kubernetes• currently Docker is the default and best supported runtime• containerd is being written to replace the relevant code in Docker • the CRI acts as an API for runtimes in Kubernetes• work on integration in https://github.com/cri-containerd/kubernetes • Kubernetes PR https://github.com/kubernetes/kubernetes/pull/43655 • containerd 1.0 milestone support in at least one runtime• likely 1.0 will be shipped with at least Docker and Kubernetes support• working with Kubernetes is an essential part of roadmap

8

containerd: view from Tim Hockin at Google

9

containerd: roadmap to 1.0Timetable is aggressive for 1.0 in 2Q 2017 but the evolution helps:• some code is reused from Docker• some is rewritten and improved based on experience• the runtime code is already in production with lots of users• focus on getting APIs clear and clean• extensible via plugins• will be supported for one year• can evolve in new directions for 2.0, ...• limit scope...

10

What is not in Containerd...

11

Limit the scope to where there is agreement, and not to constrain platforms.

• No networking• No volumes• No logging• No build

networking in containerd...

12

• No networking in containerd• https://github.com/docker/containerd/issues/362 • This is what the users of containerd wanted• Networking varies too much between platforms• continue to use CNI or other APIs as before

No networking in containerd...

13

• Provide a network namespace– Join a pre populated network namespace

• Use OCI Hooks to initialize namespace– Exec a command with the container’s state to initialize network

• Setup networking between create and start– Create container– Setup network interfaces– Start user’s process

No volume management in containerd• No consensus yet around interfaces• Storage layer can be hooked in at OCI layer• It is just a set of mounts at this level

“The Container Storage Interface (CSI) is a proposed new industry standard for cluster-wide volume plugins. This is a joint proposal from a group of us who work on Docker, Kubernetes, Mesosphere and Cloud Foundry. CSI is currently in the early draft stage, and we are seeking feedback from the community.”

14

No log management in containerd (yet)• Output streams of containers can be handled as required• Platform can arrange logging how it wishes• https://github.com/docker/containerd/issues/603 discusses changes• Possibly adding timestamps, formatting in the shim• Different use cases, eg docker run can have binary, not log output

15

No build in containerd• Use other tools for building containers• Very different concerns from runtime• The storage driver can create snapshots from a writeable filesystem

16

Ok what is in containerd then?containerd/api/services

• content• rootfs• execution• shim

Each of these are GRPC APIs

17

18

Docker storage architecture

18

Graph Driver“layers” “mounts”

Layer Store“content addressable layers”

Image Store“image configs”

Containers“container configs”

Reference Store“names to image”

Daemon

19

containerd storage architecture

19

Snapshotter“layer snapshots”

Content Store“content addressed blobs”

Metadata Store“references”

dist ctr

ConfigRootfs (mounts)

containerd: contentContent is a content addressed store

service Content {rpc Info(InfoRequest) returns (InfoResponse);rpc Delete(DeleteContentRequest) returns (google.protobuf.Empty);rpc Read(ReadRequest) returns (stream ReadResponse);rpc Status(StatusRequest) returns (stream StatusResponse);rpc Write(stream WriteRequest) returns (stream WriteResponse);

}

Content is identified via a digest, ie content hash.Status gives the status of an in progress write transaction.

20

Content Service

Write

Read

Content

Digested

containerd: rootfsservice RootFS {

rpc Unpack(UnpackRequest) returns (UnpackResponse);rpc Prepare(PrepareRequest) returns (MountResponse);rpc Mounts(MountsRequest) returns (MountResponse);

}

• Unpack a downloaded image• Prepare the root filesystem from the set of layers• Mounts returns a list of mounts to make, does not execute them

21

containerd layerscontainerd has three built in ways of layering images:• overlay• btrfs• “vfs”

These correspond to overlay and snapshotting drivers, which are the two models. The aim is to make sure the API provides support for both types, not to be comprehensive. Also a plain driver that does not use layers.

Plugins will provide additional mechanisms, eg ZFS

22

containerd: execution service ContainerService {

rpc Create(CreateRequest) returns (CreateResponse);rpc Start(StartRequest) returns (google.protobuf.Empty);rpc Delete(DeleteRequest) returns (DeleteResponse);rpc Info(InfoRequest) returns (containerd.v1.types.Container);rpc List(ListRequest) returns (ListResponse);rpc Events(EventsRequest) returns (stream containerd.v1.types.Event);

}

Fairly simple API for creating containers, and getting info about them

23

containerd: shimThe shim is a process that handles IO for a container.service Shim {

rpc Create(CreateRequest) returns (CreateResponse);rpc Start(StartRequest) returns (google.protobuf.Empty);rpc Delete(DeleteRequest) returns (DeleteResponse);rpc Exec(ExecRequest) returns (ExecResponse);rpc Pty(PtyRequest) returns (google.protobuf.Empty);rpc Events(EventsRequest) returns (stream containerd.v1.types.Event);rpc State(StateRequest) returns (StateResponse);rpc Pause(PauseRequest) returns (google.protobuf.Empty);rpc Resume(ResumeRequest) returns (google.protobuf.Empty);rpc Exit(ExitRequest) returns (google.protobuf.Empty);

}

24

containerd command lineThe command line is not the API

There are some commands for testingHowever their CLIs are unstable and may be incomplete

They can be useful for understanding what is going on, and writing tests,and trying out low level operations, eg applying layers.

25

dist: everything for images images list images known to containerd pull pull an image from a remote fetch fetch all content for an image into containerd fetch-object retrieve objects from a remote ingest accept content into the store active display active transfers. get get the data for an object delete, del, remove, rm permanently delete one or more blobs. list, ls list all blobs in the store. apply apply layer from stdin to dir rootfs setup a rootfs

26

ctr: everything for containers run run a container events display containerd events delete delete an existing container list list containers info get info about a container shim interact with a shim directly pprof provides golang pprof outputs for containerd

27

containerd end to endAs of last week there is an end to and PoC of image pull and run

dist pull docker.io/library/redis:alpinectr run --id redis -t docker.io/library/redis:alpine

28

containerd Prometheus endpointGRPC and container metrics exposed via Prometheus endpoint

# HELP container_memory_usage_usage_bytes The memory usage# TYPE container_memory_usage_usage_bytes gaugecontainer_memory_usage_usage_bytes{id="test"} 69632# HELP container_memory_writeback_bytes The writeback amount# TYPE container_memory_writeback_bytes gaugecontainer_memory_writeback_bytes{id="test"} 0# HELP container_per_cpu_nanoseconds The total cpu time per cpu# TYPE container_per_cpu_nanoseconds gaugecontainer_per_cpu_nanoseconds{cpu="0",id="test"} 180986container_per_cpu_nanoseconds{cpu="1",id="test"} 669445container_per_cpu_nanoseconds{cpu="2",id="test"} 793582container_per_cpu_nanoseconds{cpu="3",id="test"} 2.288791e+06# HELP container_pids_current The current number of pids# TYPE container_pids_current gaugecontainer_pids_current{id="test"} 1

29

containerd: Windows• Full parity for Microsoft Windows containers is planned• Not in codebase yet• Full Windows OCI support finally

And other platforms, via OCI eg runv, and some potentially direct

30

containerd• easy to use• simple design• consistent GRPC interfaces• clean design, fixes complexities from Docker• smaller component with narrower focus• easier to understand• long term interface stability• portable

31

Get involvedCurrently at

• https://github.com/docker/containerd

Will probably move since CNCF donation but there will be a redirect...

New twitter account!• @containerd on twitter

try in dockerdocker run --privileged -d dmcgowan/containerd

32

containerd weekly report

33

Docker Internals Summit @ DockerCon

• containerd in the morning with the maintainers

• Other Docker projects in the afternoon (Libnetwork, Notary, SwarmKit, InfraKit, VPNKit, DataKit, HyperKit, etc)

You don’t have to attend the conference to attend this summit on 20 April

34

THANK YOU