Oscon 2017: Build your own container-based system with the Moby project
-
Upload
patrick-chanezon -
Category
Technology
-
view
2.806 -
download
0
Transcript of Oscon 2017: Build your own container-based system with the Moby project
Patrick Chanezon, @chanezon
David Chung, @dchungsf
Mindy Preston, @mindypreston
Build your own container-based system
with the Moby project
May 2017
French
Polyglot
Platforms
Software Plumber
San Francisco
Developer Relations
@chanezon
Docker
The world needs
tools of mass innovation
A programmable Internet would be the ultimate
tool of mass innovation
A commercial product,
built on
a development platform,
built on
infrastructure,
built on
standards.
Docker is building a stack to program the Internet
Docker is building a stack to program the Internet
CE
EE
enterprise edition
Ubuntu
Fedora
Mac
Azure
CentOS
Windows 10
AWS
Debian
community edition
Ubuntu
Windows Server
Azure
CentOS
Suse
Red Hat
AWS
Oracle Linux
Orchestration
Container Runtime
OS
Infrastructure Management
Container Platform Layers
Application Services
Docker is a platform made of components
Raft StoreNode
IdentitySecrets
Routing
Mesh
Overlay
Networking
Swarm Orchestration
Engine
Application Services
12,000,000,000
11,000,000,000
10,000,000,000
9,000,000,000
8,000,000,000
7,000,000,000
6,000,000,000
5,000,000,000
4,000,000,000
3,000,000,000
2,000,000,000
1,000,000,000
Notary
runC
containerd
HyperKit , VPNKit, DataKit
SwarmKit
libcontainer
libnetwork
InfraKit
2013 2014 2015 2016 2017
1M2014
PULLS
1B2015
PULLS
6B2016
PULLS
12B2017
PULLS
linuxKit
LinuxKitA toolkit for building secure, portable and lean operating systems for containers
Taking Dockermulti-platform
“I want Docker for X”
Desktop Server Cloud
I want Docker for…
Not every platform provides a Linux subsystem
Not every platform provides a Linux subsystem
Orchestration
Container Runtime
Linux Subsystem
Infrastructure Management
Application Services
The container movement needs asecure, lean, portable subsystem
The container movement needs
a secure, lean, portable Linux subsystem.
introducing
Only works with
containers
- Smaller attack
surface
- Immutable
infrastructure
- Sandboxed system
services
- Specialized patches
and configuration
Incubator for
security innovations
- Wireguard,
Landlock, KSPP
- MirageOS type
safe system
daemons
Community-first
security process
- Linux is too big
for any one
company to
secure it
- Participate in
existing Linux
security efforts
1. LinuxKit: a SECURE Linux subsystem
- Minimal size, minimal boot time
- All system services are containers
- Everything can be removed or
replaced
2. LinuxKit: a LEAN Linux subsystem
- Desktop, server, IoT, mainframe
- Intel & ARM
- Bare metal & virtualized
3. LinuxKit: a PORTABLE Linux subsystem
Docker and Microsoft collaborate to bringLinux containers to Windows
+ +
https://github.com/linuxkit/linuxkit
Get Started with LinuxKit
MobyAn open framework to assemble specialized container systems without reinventing the wheel.
Pioneers 2013 - 2014
Production Model: open-source!
Use case: cloud native apps on Linux server
Early Adopters 2015 - 2016
Production Model: OPEN COMPONENTS
Mainstream 2017 - 2018Containers are spreading to every category of computing:
server, datacenter, cloud, IoT, desktop, mobile…
Case study:
Specializing Docker for the mainstream
Desktop Server Cloud
The open component model shows its limits…
The auto industry has solved this problem: COMMON ASSEMBLIES.
Scaling the Docker production model: share components AND
ASSEMBLIES.
It’s time to take our ecosystem to the next level…
By collaborating on components AND COMMON ASSEMBLIES.
– Library of 80+ components
– Package your own
components as containers
– Reference assemblies
deployed on millions of nodes
– Create your own assemblies
or start from an existing one
A framework to assemble
specialized container
systems without
reinventing the wheel.
Docker uses Moby for its
open-source
– Thousands of contributors,
hundreds of patches/week
– Component development
– Specialized assembly
development
– Integration tests
– Architecture design
– Integration with other projects
– Experimentation and bleeding
edge features
Docker uses Moby for its
open-source...
and so can you!
– Community-run
– Open governance inspired by
the Fedora project
– Plays well with existing
projects - no donation
necessary!
Moby and Docker
What it means for you
Moby helps you
innovate without tying
you to Docker
System BuildersDocker Users
Docker will better leverage
the ecosystem to innovate
faster for you
Moby transforms multi-month R&D projects into weekend projects.
locked-down Linux with remote attestation
Weekend project #1:
Notary
custom CI/CD stack
Weekend project #2:
Notary Registry Docker Builder
+
custom CI/CD stack + Debian+ Terraform
Weekend project #3:
Notary Docker Builder
+
Registry
“RedisOS”
Weekend project #4:
"RedisOS"for Windows
"RedisOS"for Mac
"RedisOS"for bare metal
HyperKit
bare metal
Etcd clustering on Google Cloud
Weekend project #5:
SSHD
Kubernetes on the Mac
Weekend project #6:
HyperKit
Getting Started
- Blog https://mobyproject.org/blog
- Twitter @moby
- Github moby/moby
Let’s take containers mainstream!
InfraKitA toolkit for building declarative, self-healing infrastructure.
What is it?
53
• Launched at LinuxCon, Berlin in October, 2016.
• Toolkit for building declarative, self-managing
distributed applications
• Active management with active controllers
• scaling groups, rolling updates
• monitoring / health checks
• connecting nodes to L4 / ingress
• Declarative infrastructure
Architecture
CLI
API
container orchestration
Where does it fit?
55
kubectl run nginx --image=nginx
gcloud container node-pools list --zone us-
central1-f --cluster MyWorkers
aws autoscaling update-auto-scaling-group
--auto-scaling-group-name MyWorkers
docker create service nginx …
infrakit group describe workers
az vmss create --resource-group vmss-
test-1 --name MyWorkers
container orchestration
infrastructure orchestrationinfrastructure orchestration
list, err :=
group.Controller.Describe(“workers”)
App Opscontainer orchestrationApp Ops
One console across environments
56
kubectl run nginx --image=nginx docker create service nginx …
infrakit group describe workers
container orchestration
infrastructure orchestration
list, err :=
group.Controller.Describe(“workers”)
AWS RackHDAZ GCP OneVIEWMAASKVM VMW
Cloud Ops Hardware OpsCluster Ops
Configuration
Example config file (zk.conf): Group configuration = Instance + Flavor
{"Properties": {
/* raw configuration */
}}
{"groups" : {
"my_zookeeper_nodes" : {"Properties" : {
"Instance" : {"Plugin": "instance-vagrant","Properties": {
"Box": "bento/ubuntu-16.04"}
},"Flavor" : {
"Plugin": "flavor-zookeeper","Properties": {
"type": "member","IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]
}}
}}
}}
Current Status
Support more platforms
59
• Compute:
• Bare-metal: HP OneView, MAAS, RackHD
• Public cloud: AWS, GCP
• MacOS X (HyperKit); Docker containers
• Coming soon: Azure, IBM, Digital Ocean,
Packet, libvirt
• Other resource types
• AWS - vpc, subnets, gateways, etc.
Improve usability
60
• Templates
• Complex scripts and configuration in any format;
no more escape quotes in JSON
• Fetch templates from remote repositories
• Playbooks
• CLI - flags, prompts — config driven and
dynamic
• Share “playbooks” from remote repositories
Improve core system
61
• High Availability — Swarm Mode or etcd
• New Plugin types — Metadata and Events
• Metadata: cluster-wide sysfs and reflection
• Events - publish / subscribe
• Remote client access: infrakit -H host:port to remote cluster
Road Map
Use Cases
63
• Support container orchestration
• bootstrapping + day N management
• API for cluster autoscaling
• k8s, Docker Swarm Mode
• Bare-metal + GPU provisioning
• IoT — LinuxKit integration / custom kernel
deployment
Improve usability
64
• Finalize API / Schema for 1.0
• Make it easy to consume
• Simplify setup - fewer daemons and binaries
• Embeddable / vendor API
• Sensible CLI for stable / experimental features
• Make it easy to extend / contribute
• metadata / instance plugins
• playbooks / reusable templates
• community CI / compatibility testing
• Documentation
Improve core system
65
• Provisioning of diverse resource types
• networks / proxies / load balancers
• GPU
• Stability / performance of core controllers
• Asynchronous messaging - mqtt, natsd, amqp
• Monitoring + Health check SPI
Support more platforms
66
• Direct libvirt / KVM / CUDA
• Better bare-metal / hardware ops integration
• Kernel image build pipeline — LinuxKit
Build, test, and deploy clusters from infrastructure
definitions to kernel images
Get involved
https://github.com/docker/infrakit
dockercommunity.slack.com: #infrakit
Learn More at OSCON
- Mindy Preston, Amir Chaudhry’s
“MirageOS 3: Smaller, lighter, and more transparent”
Wednesday 4:15 pm
- David Chung, Bill Farner
“InfraKit: A toolkit for infrastructure orchestration”
Thursday 11 am
THANK YOU