Being a Moby maintainer

Copyright©2017 NTT Corp. All Rights Reserved.

Akihiro Suda ( @_AkihiroSuda_ )

NTT Software Innovation Center

Being a Moby maintainer

Docker Tokyo (Nov 2, 2017)

https://slideshare.net/AkihiroSuda

https://slideshare.net/AkihiroSuda

2Copyright©2017 NTT Corp. All Rights Reserved.

• Software Engineer at NTT

• github: @AkihiroSuda

• Twitter: @_AkihiroSuda_

• Docker Moby core maintainer (github.com/docker/docker moby/moby)

• In April 2017, Docker [ as a project ] transited into Moby.

• Now Docker [ as a product ] has been developed as one of downstream of Moby.

: ≒ :RHEL Fedora

Who am I

github.com/AkihiroSuda

https://twitter.com/_AkihiroSuda_

https://github.com/docker/docker


• BuildKit initial maintainer (github.com/moby/buildkit)

• Next-generation `docker build`

• Executes DAG vertices of Dockerfile-equivalent concurrently

• Soon: cache-aware distributed mode

• containerd maintainer (github.com/containerd/containerd)

• Industry-standard container runtime

• Can be used as a Docker-replacement for Kubernetes

Who am I

https://github.com/moby/buildkit

https://github.com/containerd/containerd


• What is the Moby Project

• Recent updates in the Moby Project

• How I became a maintainer (and how you can!)

Agenda


What is the Moby Project?


• A collaborative project for the container ecosystem to assemble

container-based systems


runc

BuildKit

Moby registry

DataKit

VPNKit

HyperKit

Moby engine(dockerd)Moby tool

libnetwork

libentitlement


Relationship between Moby and Docker (and Balena)

Docker Community Edition Docker Enterprise Edition

Balena: Moby-based container engine for IoT, by Resin.io

+ Support, GUI..

Add your own downstream here...

?

Third parties

Docker, Inc.'s products

Similarity


• `dockerd` command is part of Moby• Will be renamed to `moby-engine` soon

• `docker` command is NOT part of Moby• Because UX is basically out of scope of Moby

• Solely maintained by Docker, Inc. But still opensource.

• Desktop and cloud installers are NOT part of Moby(Docker for Mac / Windows / AWS / Azure)• Docker, Inc. 's proprietary software at the moment

• Dockerfile is likely to be removed from Moby... But no worry!• Moby BuildKit provides a new low-level instruction set

Some Docker components are not included in Moby


• Announcement at DockerCon EU (October 17, 2017)

Recent updates in Moby and Docker

Kubernetes

Docker

containerd

Docker

Kubernetes

containerd

API translator (unreleased)

`docker` CLI

can be used for

managing k8s

Container Runtime Interface (CRI)

k8s no longer

depends on

Docker


• Announcement at DockerCon EU (October 17, 2017)


https://blog.docker.com/2017/10/kubernetes-docker-platform-and-moby-project/

Swarm is still kept

(User can choose either one)

https://blog.docker.com/2017/10/kubernetes-docker-platform-and-moby-project/


• This was not a surprise, because Docker, Inc. and the Moby

community has been already working on Kubernetes-related stuff

for a year


https://blog.mobyproject.org/moby-and-kubernetes-bf888ab31e38

https://blog.mobyproject.org/moby-and-kubernetes-bf888ab31e38


containerd: Industry-standard container runtime

• Simpler architecture than the "monolith" of Docker

• containerd is "a la carte" of well-decoupled subsystems (runtime, image, snapshot..)

• Can be used as a Docker-replacement for Kubernetes (and so on)

• CRI-containerd, the glue module for k8s is mainly maintained by Google employees

• More collaborative than the past Docker in the pre-Moby era

• Docker, Inc. donated containerd to Cloud Native Computing Foundation (CNCF)

runc

containerd v1.0

CRI-containerd

KubernetesKubernetes

Docker (Moby)

runc

containerd v0.2

Docker (Moby)

Docker 18.XX (unreleased)


• Assembles VM images for running certain service like Docker

• Now officially supports Kubernetes as well (with CRI-containerd)

• Usecases are not limited to Docker/Kubernetes

• e.g. "RedisOS" without Docker nor Kubernetes

• Everything is containerd container

• No systemd nor SysV init

• Everything is immutable by design

• When you need to update or customize your LinuxKit VM instances, you just

dispose the existing instances and create new ones (as in Docker containers!)

LinuxKit: Toolkit for custom Linux distribution

LinuxKit

namespace

Kubernetes

namespace

dhcpd

ntpd

kubelet

cri-containerd

e.g., nginx

containerd


• Deploys distributed system like Docker Swarm-mode to IaaS like EC2

• Self-healing

• Recreate instances if the actual state differs from the desired state

• Now supports deploying Kubernetes as well

(implemented by Yuji Oshima)

• InfraKit maintainer

• My colleague at NTT

InfraKit: Toolkit for infrastructure orchestration

Infrakit LinuxKit LinuxKit

EC2 EC2

LinuxKit

EC2

Docker Swarm-mode / Kubernetes


• Compiles Dockerfile to LLB DAG

• LLB: low-level build instruction set

• DAG: directed acyclic graph

• LLB DAG allows concurrent build

• Third party languages could be compiled to LLB DAG as well

BuildKit: next-generation `docker build`

Compile

Dockerfile

LLB DAG

Third party languages

docker-image://alpine

Image

git://foo/bardocker-image://gcc

Run("apk add ..")Run("make")


• Distributed mode (on Kubernetes) is on plan

• Workers reports LLB DAG vertex cache info and performance stats to masters

• Master assigns a vertex job to the worker which seems the best (just heuristic)

• Designed to be stateless as much as possible for ease of deployment


Master

Master

Master

LBClient

Worker

Worker

Worker


Even useful for non-container

usecases, as a generic

concurrent / distributed

compiler toolkit



libentitlement: high-level permissions and security profile

https://www.slideshare.net/Docker/moby-and-kubernetes-entitlements



libentitlement: high-level permissions and security profile




• Previously, Solomon Hykes (CTO, Docker, Inc.) was the BDFL

• Benevolent Dictator For Life

• Now "Technical Steering Committee" is being established (in this

November) as the replacement for the BDFL role

• A single company cannot hold more than 1/3 seats

Moby governance


• Some components under the Moby umbrella belong to other

organizations

• containerd, Notary: CNCF (Cloud Native Computing Foundation)

• InfraKit is also proposed to CNCF

• runc: OCI (Open Containers Initiative)

• Moby Technical Steering Committee will help cross-project and

cross-organization collaboration

Moby Project, CNCF, and OCI


• Maintainers (aka "committers" in other FLOSS communities)

• Can approve other contributors' pull requests (with 2 LGTMs usually)

• Can manage GitHub issues

• Can add and remove other maintainers (with 66% approval vote)

• Elected from active contributors who:

• Send pull requests

• Bug fix

• Enhancement

• New feature

• Review other contributors pull requests

• Triage GitHub issues

Who are maintainers and how they are elected


• Began contribution to Docker in December, 2015• I was working on some fault injection tool (github.com/osrg/namazu), and got

stuck in "false ZooKeeper bug" due to AUFS hang-up: #18180

• This is not a bug of Docker but mainly tracked in Docker community

• Became a Docker maintainer in November, 2016• Docker Moby in April, 2017

• (I think) Mainly contributed to filesystem issues• Both AUFS and overlayfs have some stability and compatibility issues

• Also proposed some new features

• `docker network prune` (merged): #27525

• introspection mount (procfs-like stuff for containers. unmerged yet): #24893, #26331

• TCP port forwarder (withdrawn): #26365

• ...

How I became a Docker/Moby maintainer

https://github.com/osrg/namazu

https://github.com/moby/moby/pull/18180






How I became a Docker/Moby maintainer

JVM processes in Docker were extremely unstable due to an

AUFS issue.

The issue was very critical for many users, but very hard to

debug because of non-determinism.

Actually, it was not me who firstly identified

the cause of the issue (Also, I didn't wrote

the patch to fix the issue in the fact)

But I made some demonstration

tools for confirming the cause of

the issue

I made some reports to AUFS community,

and the AUFS maintainer (not me) fixed the

issue

Created "cheat sheet" for Linux

distribution-specific workarounds

Encouraged Linux distributors to

apply the AUFS patch

Created comprehensive list of filesystem

issues

(github.com/AkihiroSuda/issues-docker)

https://github.com/AkihiroSuda/issues-docker

Began code contribution

(Details are out of scope of this talk)

Discussed with maintainers in person

(Using mobile phones!)

http://www.publickey1.jp/blog/16/docker_enginenttdocker.html

"Suda at NTT became a maintainer of

Docker Engine"

http://www.publickey1.jp/blog/16/docker_enginenttdocker.html


• Initial maintainer from the beginning of the project (2017 summer)

• Initially I proposed DAG-based builder (but without idea of LLB)

• Coincidently, Tõnis Tiigi (Docker, Inc.) was planning similar but

even better idea, which turned into BuildKit

• Tõnis invited me to an initial maintainer of BuildKit

How I became a BuildKit maintainer


• Began contribution in December, 2016

• Became a maintainer in September, 2017

• Mainly contributed to filesystem and image issues

• Though contribution to Docker / Moby, I found filesystem issues are hard to

maintain

• I needed to reform containerd interfaces and data formats for my experimental OCI

Image extension

• FILEgrain: transport-agnostic, fine-grained content-addressable container image layout

(github.com/AkihiroSuda/filegrain)

How I became a containerd maintainer

https://github.com/AkihiroSuda/filegrain


• Collaborativeness (the most important)

• Comprehensiveness

• Issue analysis, Bug-fix, enhancement, feature addition...

• But when you plan to add a big feature, please coordinate with maintainers in GitHub

issue or Slack before opening a PR!

• Continuity

• One-shot contribution is always welcomed, but maintainership requires continuous

activity

• Number of git commits and LOCs are not so important

And how you can become maintainers! (my personal view)


• Moby engine

• Recently integrated containerd v1.0 runtime. We need to make sure there is no

regression.

• The next step is to integrate containerd v1.0 snapshot subsystem. (Much harder for

compatibility)

• BuildKit

• Dockerfile2LLB compiler is not stable. Testing is highly welcomed.

• Design for distributed mode is still under discussion.

• containerd

• Adding tests and performance optimizations are welcomed.

• And more!

Good chances to contribution (my personal view)


• The Moby Project is getting more collaborative

• You can contribute and become a maintainer as well!

Recap

https://blog.docker.com/2017/04/introducing-the-moby-project/

https://blog.docker.com/2017/04/introducing-the-moby-project/

Being a Moby maintainer

Software

Transcript of Being a Moby maintainer