Docker HubPast, present and future

Ken Cochrane@KenCochrane

bc Wong@bcwalrus

We are engineers working on Docker Hub

Docker Hub

Docker Hub (hub.docker.com) is a cloud registry service forsharing application and automating workflows.

Major features:• Public and Private Docker repositories• Official Repositories• Collaborators, Organizations and Groups• Automated builds• Webhooks

3

Docker Hub - A brief history

Hub by the Numbers

• 240,000 Users

• 150,000 Repositories

• 1,250,000 Automated Builds

• 500,000,000 Pulls

5

Hub Growth (disk space)

6

Hub Growth (bandwidth)

7

Growth of a distributed application

Docker Hub: This is your life

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub: This is your life (cont.)

Docker Hub Timeline

18

Docker Hub - More than just a registry

Organizations & Groups

• Best suited for businesses• Collaborate on repos with members of your organization• Fine-grain permissions• Add users to groups• Public and Private repos

20

Official Repositories

• 80 Official Repos that you can use to build your Docker applications• Repos are updated and maintained by Docker

21

Private Repositories

• Not searchable or browsable• Can add collaborators to push or pull• Only you or a collaborator can access• 1 free private repo with your docker hub user account

22

Coupon for 3 free months of the micro plan

dockercon15

Automated Builds

• Link to your Github or Bitbucket accounts• Keep your source code in sync with your docker repos• You write the code, we handle the build, and keep your Docker hub

repos up to date.• Public and Private repos

23

Repository Links

What if your base ubuntu:14.04 image is updated for a security fix?

• Link your automated build repository to another repository• Allows you to keep your automated build up to date

24

Build Triggers

• Trigger an automated build from an external system• Useful for integrating with CI systems

25

Webhooks

• Get notified when a docker repo is updated• Ability to chain together webhooks to create a build pipeline• Combine with a continuous integration and deployment solution to

easily move your application from development to server.

26

And much more ...

• Search• Stars• Comments• Notifications

27

Upcoming Features

• Show image sizes• GitLab support for automated builds• Dynamic tagging for automated builds

28

Docker Hub 2.0 (beta)

Hub 2.0

• New, Easier to use UI• Re-architected, isomorphic javascript application based on react.js,

react-router and fluxible• A modern, modular approach using ES2015 enabling faster iterations

30

Data stores

Web UI

API Gateway

Postgresql Redis KafkaRabbitMQElastic Search

Micro Services

S3

New Architecture

Live demo time: Hub 2.0hub-beta.docker.com

Left brain: What could go wrong?Right brain: Some people never learn.

Give us feedback please

github.com/docker/hub-beta-feedback

33

Docker Hub - Performance

Docker Hub Performance - push/pull

35

Architecture Upgrade - Registry v2

• Registry v2 protocol is far superior to v1• Supported by Docker >=1.6• Content-addressable manifest• More secure

36

80% fewer requests

60% less bandwitdth

Architecture Upgrade - Disentangle

• Clean up inter-service dependencies:

• Hub 2.0 uses a gateway with finer grain services.• Reduces cascading failures.

37

Infrastructure Upgrade

• Aka “throw money at the problem”• The difficult parts are when (i.e. monitoring), and the execution.• E.g. database upgrade:

38

External ping latency (worldwide average)DB response time (before vs after)

Better Service Isolation

39

Automated Builds (v1 current)

Architecture• A set of long-running build servers w/o auto-scaling• One build per server at a time (docker in docker)• Thousand’s of builds per day

40

Challenges• # of builds <= # of servers, which

means pending builds• Security concerns• Build server issue == lost capacity

Automated Builds (v2 upcoming)

Speed and efficiency• Auto-scale on-demand builds• Per-user queuing• Better scheduling

Security• Use VM for isolation between users• TLS everywhere• Ephemeral build servers

41

For those about to rock ...

Scalability & Reliability Work: Myth vs Reality

43

Need a team that can do the first 80% and the last 20%.Need a culture that understands quality.

Hey, it’s still fun

Why is my sink not draining?

From our post-mortems & bug reports:• DB capacity• ORM layer makes slow queries• Networking flakiness• RabbitMQ failures (and we didn’t understand its failure models)• Supervisor failed to restart appserver, but NGINX keeps accepting calls• Expired intermediate cert• Docker btrfs driver bug• ...

Know when to tighten a bolt, and when to get a new sink.

44

High Velocity Cycle (invest in tooling)

45

Monitoring& logging(what’s slow)

Features & fixes

QA, scale test,perf regression test(prove that things work)

Future Performance Work

• Continue to eliminate bottlenecks as Hub grows• Speed up push/pull outside the US• Scalable pipeline into data warehouse

46

Credits: The Hub Team

Ryan AbramsChris BiscardiKen CochraneJohn CostaRoberto HashiokaDaniel HiltgenNathan HsiehNick KralyAdrian KreherToli KuznetsDustin Lacewell

47

Nick LangSean LiAmy LindburgMarcus MartinsChinmayee NirmalAtul PhadnisMario PonticelloArunan RabindranJosh SouthJusten Stepkabc Wong

Credits: The Hub Team

Questions?Remember to check out hub-beta,

and send us feedback:github.com/docker/hub-beta-feedback