Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 ·...

35
Docker Volumes with BeeGFS Jasper Lievisse Adriaanse Docker: Linux Containers to Optimise IT Infrastructure for HPC & BigData [June 23, 2016]

Transcript of Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 ·...

Page 1: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

Docker Volumes with BeeGFS

Jasper Lievisse Adriaanse

Docker: Linux Containers to Optimise IT Infrastructure for HPC & BigData

[June 23, 2016]

Page 2: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Agenda]

Introduction

Docker

VolumesIntroductionTypesPlugins

BeeGFSBeeGFSDockerized stackBeeGFS plugin

Demo

Q&A

Page 3: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Introduction] Who am I?

CTO for RedCoolBeansOpenBSD developer

[email protected] jasper_la , jasperla

Page 4: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Introduction] RedCoolBeans

I Secure Docker stacksI CargOS, Dockerlint, Crane

RedCoolBeans @RedCoolBeans RedCoolBeans.com and CargOS.io

Page 5: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Docker] Adoption Hurdles

Two primary challenges:I Networking

I Storage

Page 6: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Docker] Adoption Hurdles

Two primary challenges:I NetworkingI Storage

Page 7: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Docker] Networking

I LinksI Multi-host setups

I Ambassador pattern

Page 8: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Docker] Networking (cont.)

Two “solutions”:I WeaveI docker-compose

Docker 1.9 network subcommand

Page 9: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Docker] Data persistence

Containers are ephemeral, data is forever (sort of)Not all containers are stateless

I DatabasesI CMS uploadsI …

Page 10: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Purpose of volumes

I Store data outside of a container

I Share data between containersI Backup/replicationI Utilize external/clustered filesystems

Page 11: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Purpose of volumes

I Store data outside of a containerI Share data between containers

I Backup/replicationI Utilize external/clustered filesystems

Page 12: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Purpose of volumes

I Store data outside of a containerI Share data between containersI Backup/replication

I Utilize external/clustered filesystems

Page 13: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Purpose of volumes

I Store data outside of a containerI Share data between containersI Backup/replicationI Utilize external/clustered filesystems

Page 14: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes]

Data volumesI docker run -v /data:/data […]

Data volume containersI docker run --volumes-from data-container […]

Page 15: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Types of volumes

Data volumes on host (host volumes)I docker run -v /docker/db:/db […]

Page 16: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Types of volumes (cont.)

Data volumesAnonymous volume

I docker run -v /data […]I Data lifetime matches container’s

Named volumeI docker run -v datavolume:/data […]I Data lifetime exceeds container’s

Page 17: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Types of volumes (cont.)

Data volumesAnonymous volume

I docker run -v /data […]I Data lifetime matches container’s

Named volumeI docker run -v datavolume:/data […]I Data lifetime exceeds container’s

Page 18: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Types of volumes (cont.)

Data volume containersI docker run --volumes-from data-container […]I Store data in containers

Page 19: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Security implications

I No restrictions on sharingI SELinux-labelingI Boot2docker situation

Page 20: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Plugins

I docker volume subcommand/APII Arbitrary backend providerI Plugins for:

I NFS, CIFS, EFSI GlusterFS, IPFS

I BeeGFS

Page 21: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Volumes] Plugins

I docker volume subcommand/APII Arbitrary backend providerI Plugins for:

I NFS, CIFS, EFSI GlusterFS, IPFSI BeeGFS

Page 22: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS]

I Open source, highly performant cluster filesystem

I Easy to setupI Excels on commodity hardwareI Developed by Fraunhofer institute

Page 23: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS]

I Open source, highly performant cluster filesystemI Easy to setup

I Excels on commodity hardwareI Developed by Fraunhofer institute

Page 24: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS]

I Open source, highly performant cluster filesystemI Easy to setupI Excels on commodity hardware

I Developed by Fraunhofer institute

Page 25: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS]

I Open source, highly performant cluster filesystemI Easy to setupI Excels on commodity hardwareI Developed by Fraunhofer institute

Page 26: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS] Overview

Figure: Architecture overview (source: beegfs.com)

Page 27: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Dockerized stack] Services

I ManagementI MetadataI Storage

I Client

Page 28: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Dockerized stack] Services

I ManagementI MetadataI StorageI Client

Page 29: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Dockerized stack] Availability

I DockerHubI github.com/RedCoolBeans/docker-beegfs

Page 30: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS Plugin]

I github.com/RedCoolBeans/docker-volume-beegfsI Available for:

I RedHat (+CentOS, &c)I Debian (+Ubuntu, &c)

Page 31: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS plugin] Requirements

I Docker 1.8I Plugin service runningI Kernel module loadedI BeeGFS share mounted

Page 32: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS plugin] How does it work?

I Bridge to Docker API (docker volume)I Maps volume name to on-disk nameI Checks underlying filesystem type

Page 33: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[BeeGFS plugin] Use cases

I Multi-host Docker clusterI Redudant storageI (Live-)migrationI CI/CD

Page 34: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Demo]

Demo

Page 35: Docker Volumes with BeeGFS - QNIBqnib.org/data/isc2016/5_docker_volumes.pdf · 2019-09-08 · Introduction Docker Volumes Introduction Types Plugins BeeGFS BeeGFS Dockerizedstack

[Q&A]

Questions?


Docker Machine & Docker Swarm

Docker Machine & Docker Swarm

Docker Basics - docs.adfinis-sygroup.ch · Check your docker version docker version Check the available docker options docker Your first hello-world container docker run hello-world

Docker Basics - docs.adfinis-sygroup.ch · Check your docker version docker version Check the available docker options docker Your first hello-world container docker run hello-world

docker run docker service is the new · Getting Started with Docker Clustering Mike Goelzer / mgoelzer@docker.com / @mgoelzer Docker Inc. docker service is the new docker run docker

docker run docker service is the new · Getting Started with Docker Clustering Mike Goelzer / [email protected] / @mgoelzer Docker Inc. docker service is the new docker run docker

Dell EMC Ready Solutions for HPC BeeGFS Storage...Dell EMC Ready Solution s for HPC BeeGFS Storage2 (referred to as Dell EMC BeeGFS Storage Solution in the rest of the document), which

Dell EMC Ready Solutions for HPC BeeGFS Storage...Dell EMC Ready Solution s for HPC BeeGFS Storage2 (referred to as Dell EMC BeeGFS Storage Solution in the rest of the document), which

Hands on Docker - iedu.us€¦ · Docker orchestration using CoreOS and fleet CoreOS Architecture Networking, security and API’s Docker networking Docker security Docker Remote

Hands on Docker - iedu.us€¦ · Docker orchestration using CoreOS and fleet CoreOS Architecture Networking, security and API’s Docker networking Docker security Docker Remote

Containerisation with Docker ...Docker Swarm: Produces a single, virtual Docker host by clustering multiple Docker Swarm Docker hosts together. It presents the same Docker API; allowing

Containerisation with Docker ...Docker Swarm: Produces a single, virtual Docker host by clustering multiple Docker Swarm Docker hosts together. It presents the same Docker API; allowing

Docker 101: Introduction to Docker

Docker 101: Introduction to Docker

events.static.linuxfound.org · 2017-12-14 · OpenStack Cartridge Instance n create/destroy instances Cartridge Instance 2 os hardware Pocker Docker Docker Docker Docker Docker Docker

events.static.linuxfound.org · 2017-12-14 · OpenStack Cartridge Instance n create/destroy instances Cartridge Instance 2 os hardware Pocker Docker Docker Docker Docker Docker Docker

Docker for the UseR - Microsoft · docker pull [user/repo] docker build [directory] -t [tag] docker run [image] -p [ports] -v [volumes] [command] -d docker stop [container] Start

Docker for the UseR - Microsoft · docker pull [user/repo] docker build [directory] -t [tag] docker run [image] -p [ports] -v [volumes] [command] -d docker stop [container] Start

Docker on Docker

Docker on Docker

Docker Workshop - Orchestrating Docker Containers

Docker Workshop - Orchestrating Docker Containers

Whitepaper transtec & BeeGFS...transtec is a registered gold partner of ThinkParQ and installed the fi rst Petabyte BeeGFS system worldwide. > About This document presents a summary

Whitepaper transtec & BeeGFS...transtec is a registered gold partner of ThinkParQ and installed the fi rst Petabyte BeeGFS system worldwide. > About This document presents a summary

Hands on BeeGFS - MathriceHands on BeeGFS Table of Contents Main objectives Practical Work 1 Practical Work 2 Practical Work 3 BeeGFS Concepts Practical Work 1 OpenNebula VM Startup

Hands on BeeGFS - MathriceHands on BeeGFS Table of Contents Main objectives Practical Work 1 Practical Work 2 Practical Work 3 BeeGFS Concepts Practical Work 1 OpenNebula VM Startup

Docker Swarm: Docker Native Clustering

Docker Swarm: Docker Native Clustering

Real virtual environments without virtualenv · Service discovery – Compose – Swarm – Consul – etcd – ZooKeeper. Secrets – env vars – volumes with secrets – docker-vault

Real virtual environments without virtualenv · Service discovery – Compose – Swarm – Consul – etcd – ZooKeeper. Secrets – env vars – volumes with secrets – docker-vault

Installation, container management, Docker - iol.unibo.it · • Docker containers are built from Docker images. By default, Docker pulls these images from Docker Hub, a Docker registry

Installation, container management, Docker - iol.unibo.it · • Docker containers are built from Docker images. By default, Docker pulls these images from Docker Hub, a Docker registry

Xen Containers: Better way to run Docker Containers5 Docker Containers Running • docker run/create/stop Building • docker build Packaging • docker push/pull/commit Docker –a

Xen Containers: Better way to run Docker Containers5 Docker Containers Running • docker run/create/stop Building • docker build Packaging • docker push/pull/commit Docker –a

Flocker Documentation - Read the Docs · Flocker Documentation, Release 0.0.6-dirty 1.2.3Flocker - Application State •Flocker manages ZFS filesystems as Docker volumes. It attaches

Flocker Documentation - Read the Docs · Flocker Documentation, Release 0.0.6-dirty 1.2.3Flocker - Application State •Flocker manages ZFS filesystems as Docker volumes. It attaches

Technology Testing at CSCS including BeeGFS Preliminary ... · Technology Testing at CSCS including BeeGFS Preliminary Results Hussein N. Harake ... BeeGFS Case Study. ... • Use

Technology Testing at CSCS including BeeGFS Preliminary ... · Technology Testing at CSCS including BeeGFS Preliminary Results Hussein N. Harake ... BeeGFS Case Study. ... • Use

Docker Meetup Bangalore - Docker + Openstack

Docker Meetup Bangalore - Docker + Openstack