Docker for HPC in a Nutshell

Andreas Schmidt | @aschmidt75

Docker: User Friendly Application & Service Containers for HPC Environments Docker In A Nutshell

2 16.07.15 Cassini Consulting

vs. Pets Cattle

16.07.15 Cassini Consulting 6

Technical Perspective

Developer Perspective

Security

Ecosystem & Partners

Security

Docker Daemon

Docker Engine core components

Containers

Images

Containers

Docker Daemon

Docker Engine core components

Containers

Images

Containers

Docker Client

The Docker workflow & commands

Registry

Docker- file

Registry

Docker- file

push pull

Registry

Container

Docker- file

push pull

Registry

Container

Docker- file

push pull

commit

Registry

Container

Docker- file

push pull

commit

start, stop, ...

Registry

Container

Docker- file

push pull

commit

start, stop, ...

Registry

Container

Docker- file

✖ ✖

push pull

commit

rm rmi

start, stop, ...

Registry

Container

Docker- file

External Tarball

✖ ✖

push pull

commit

rm rmi

start, stop, ...

export

Docker Engine technology foundation

Isolation through Kernel Namespaces Linux Capabilities Resource limitation through Linux control groups Filesystem isolation, Copy-On-Write & Union FS

Docker Engine technology foundation

21 16.07.15 (*) https://www.docker.com/whatisdocker

Virtual Machines vs. Containers

22 16.07.15 (*) https://www.docker.com/whatisdocker

Security

From Development to Production: Challenges ahead!

What Docker gives to developers

Easy sandbox approach

Easy way to deliver software

Dev / Prod parity

Easy Sandbox approach

§  Create build and test environments

§  Choose Libraries and Framework Dependencies, per application

§  Deal with incompatible version mismatches (Ruby 1.9 vs. Ruby 2.1, including libs, bundler, version switchers, ...)

§  Lightweight alternative to using virtual machines

Easy Sandbox approach

A Docker Image contains its own userland libraries and binaries

§  separated from other images

§  take exactly the versions of libraries you need

§  leave out things you do not need

§  reproducible, lightweight, easily testable

§  look at it as a unit of delivery

Code Deliverables (i.e. RPM packages) are not runnable.

An installed instance (i.e. in a VM) is hard to transport.

https://twitter.com/joyent/status/565243828718678016

Dockerfile as a Contract between Development and Operations

what to base from

set environment params

prepare the image, i.e. install something, configure it

describe the interface

what to run

Dockerfile as the blueprint for reusable building blocks

what to base from

Redis is an official repository at dockerhub

Dockerfile as the blueprint for reusable building blocks

what to base from

Redis is an official repository at dockerhub

Pave the way for Dev/Prod parity

§  From Applications to Systems

§  Describe not only compute, but also storage and networking.

§  Example: docker-compose

Security

Is Docker secure?

Let's ask this question more specific.

Is the docker daemon secure? Are images transported securely? Are images built in a secure fashion? Are containers as secure as virtual machines? Are application processes more or less secure when containerized?

Are containers as secure as virtual machines?

Control group-separated, chroot-like, namespaced resources, running on a shared kernel.

Application Containers Virtual Machines

Virtualized pieces of hardware, running its own kernel with process/user/network spaces separated on hypervisor level.

Are application processes more or less secure when containerized?

Definitely more secure, if "used properly(*)" Docker Container ~ Application process, ideally a single process, without management daemons Smaller attack surface Namespaced process, network, FS mounts, ...: ~ Application cannot see "the outside OS world" Reduced Linux Capabilities, can be fine tuned Additional isolation mechanisms at hand: SELinux Type Enforcement, AppArmor profiles, Libseccomp System Call Filtering

38 16.07.15 Cassini Consulting (*) http://container-solutions.com/is-docker-safe-for-production

Docker Hardening – Docker Security Benchmark

§  Extensive Guide on hardening docker hosts, images and containers, including checks

§  Automated tools are in development

§  benchmarks.cisecurity.org

Security

Tooling around Docker

Where to run Specialised Operating systems

Where to pull images from Registries (Private, On Premise, ...)

How to operate it Orchestration, Scheduling, Management, Monitoring

From Infrastructure to Applications

Platform-As-a-Service

How to build containers Config Management, Developer Tools

Technical topics Networking, Security, Storage

Competitors & Container Runtime Alternatives

LXC + LXD

CoreOS Rocket (rkt), Application Container Spec (appc)

Open Container Project (www.opencontainers.org)

Technical Perspective Developer Perspective

Technical Perspective Developer Perspective Security

Technical Perspective Developer Perspective Security Ecosystem & Partners

Technical Perspective Developer Perspective Security Ecosystem & Competitors Thank you!

Docker for HPC in a Nutshell

Technology

Transcript of Docker for HPC in a Nutshell

R and High-Performance Computing · High-Performance Computing (HPC) In a nutshell, HPC is working with · (much) more data than single machine can handle · (many) more processors

Docker 101 - techccu.csie.iotechccu.csie.io/2015/slides/frank.pdf · Docker Basics - CLI Docker client docker version docker info docker search [keyword] docker push/pull/commit docker

Docker, VMs, and Cloud Architectures for HPC › ... › 08 › Keahey_330aug1_… · • Experiments comparing KVM and Docker • Work by Yuyu Zhou and Balaji Subramaniam • Chameleon

HPC Visualization Containers - Nvidia · docker2 for your operating system and Docker version. For a script to install nvidia-docker2, see Installing NVIDIA Docker 2.0. If you already

Contain This, Unleashing Docker for HPC · Contain This, Unleashing Docker for HPC 1 ... the"power"of"Docker" ScienceatScale%% % Petascale(to(Exascale(Simula0ons(SciencethroughVolume%%

Exploring Emerging Technologies in the HPC Co …...Presentation in a nutshell Our community expects major challenges in HPC as we move to extreme scale –Power, Performance, Resilience,

State-of-the art containers & hypervisorsmmc.geofisica.unam.mx/.../VirtualizacionEnLinuxCon-HPC-VPS/Cont… · Docker container isolation I Force Docker container to only use some

Docker Docker - Docker Security - Docker

CQRS, ReactJS, Docker in a nutshell

events.static.linuxfound.org · 2017-12-14 · OpenStack Cartridge Instance n create/destroy instances Cartridge Instance 2 os hardware Pocker Docker Docker Docker Docker Docker Docker

What drives Docker in Non-HPC? - QNIBqnib.org/data/isc2016/2_docker_drivers.pdf · 1. Docker SWARM Simple to grasp extends and builds on Docker API 2. Datacenter Orchestration Derived

Docker : devops, shared registries, HPC and emerging use cases

The new Docker networking put into action to spin up a ...hpcadvisorycouncil.com/.../pdf/Kniep.DockerNetworkingSlurm.Gaikai.pdf · Multi-host containerised HPC clusterss The new Docker

Containers for Science - HPC Advisory Council · So what about Docker!? Docker is the most well known and utilized container platform Designed primarily for network micro-service

ETP4HPC IN A NUTSHELL in a... • office@etp4hpc.eu ETP4HPC IN A NUTSHELL Building a globally competitive European world- class HPC technology value chain 13 November 2017What you

Quentin Le Burel CIUK 2016 - 193.62.125.70193.62.125.70/CIUK-2016/ClusterVision.pdf · Full HPC performance using the optional Docker-based application containerization Parallel filesystem

High Performance Computing Melbourne | HPC Solutions Melbourne · NVIDIA THIRD PARTY ACCELERATED SOLUTIONS MAP o '"Blazing graphistry CONTAINERIZATION TOOL NVIDIA Docker GPU DRIVER

CQRS, React, Docker in a Nutshell

DOCKER 101 - DeveloperMarch€¦ · -Docker Machine -Docker Compose -Docker Stack -Docker Swarm -Docker Hub. INSTALLATION. INSTALLATION. WORKFLOW. client Dockerfile Image Registry

HTCondor, Docker, iRODS and HPC for every scientist with the ...