ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

Using Docker Containers to Improve Reproducibility in Software and Web Engineering

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

2

Jürgen Cito PhD @ UZH

Vincenzo Ferme PhD @ USI

The speakers

3

This tutorial…

Conceptual, more abstract

notion of Reproducibility

Concrete instructions

to aid Reproducibility in

Research & Industry

4

Agenda

Part I:What is Reproducibility? A theoretical perspective

4

Agenda

Part I:What is Reproducibility? A theoretical perspective

Part II:Docker to Aid Reproducibility Basics and Case Study 1 (Research Project Case Study)

4

Agenda

Part I:What is Reproducibility? A theoretical perspective

Part II:Docker to Aid Reproducibility Basics and Case Study 1 (Research Project Case Study)

Part III:Docker to Aid Reproducibility Advanced Topics and Case Study 2 (Industrial Case Study)

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

Part I - What is Reproducibility?

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

ICWE 2016, Lugano

6

Reproducibility is the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently.

Reproducing an experiment is called replicating it.

What is Reproducibility?

No research paper can ever be consideredto be the final word, and the replication and corroboration of research results is key to the scientific process.

7

What is Reproducibility?

Repeatability of a certain process in order to establish a fact or the conditions under which we are able to observe the same fact*

A process to share methods, and describe the environment, in order to recreate results.

* Mockus et al. “Experiences from replicating a case study to investigate reproducibility of software development.”

8

Scientific Process

Experiment Design

Data Collection

Data Analysis

Results Interpretation

Hypotheses

Reproducibility

9

What is Reproducibility?

Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge

9

What is Reproducibility?

Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge

Controlling environment > Execution environment> Dependencies

9

What is Reproducibility?

Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge

Controlling environment > Execution environment> Dependencies

Providing data > Ability to interpret data > Computational analyses

9

What is Reproducibility?

Establishing facts > Steps (or a method) to establish the fact > Sharing computational knowledge

Controlling environment > Execution environment> Dependencies

Providing data > Ability to interpret data > Computational analyses

Low barriers to replicate > Comprehensible results> Ease of achieving replication

10

What is Reproducibility in SE/WE Research?

Algorithms / Computational Analyses

Developed Tools or Prototypes

Quantitative Evaluations

+ internal knowledge of the necessary process to derive/establish results

11

In Software Engineering Research:- FSE’15/16, MSR’15/16

In Programming Languages Research:- PLDI, POPL, OOPSLA

Artifact Evaluation & Replication Packages

12

Current State of Sharing Artefacts

Researcher’s website

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

Part II - Research Project Case Study

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

ICWE 2016, Lugano

14

Case Study 1: ChangeDistiller

https://bitbucket.org/sealuzh/tools-changedistiller

Research Project

15

Why is reproducibility hard?

Why does it fail?

16

Reasons for Failed Reproducibility (1/2)

*https://twitter.com/ianholmes/status/288689712636493824

17Source: Collberg et al., “Measuring Reproducibility in Computer Systems Research”, http://reproducibility.cs.arizona.edu/tr.pdf

Reasons for Failed Reproducibility (2/2)

18

Case Study 1: ChangeDistiller

https://bitbucket.org/sealuzh/tools-changedistiller

Research Project

- Developed 2006-2009 How many Java versions have we passed?

- Dependencies defined in a Maven pom file Are they all still available in the repository?

- How does analysis in ChangeDistiller work?What is the entry point?

19

Challenges in Reproducibility

19

> No standard way of describing experiments, environments, (derived) data, and workflows

Challenges in Reproducibility

19

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis

Challenges in Reproducibility

19

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis

> The experimental nature of the research code and ecosystems makes it often hard to build

Challenges in Reproducibility

19

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis

> The experimental nature of the research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

Challenges in Reproducibility

19

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis

> The experimental nature of the research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Infrastructure for storage and distribution

Challenges in Reproducibility

20

… to the rescue

21

Docker Containers to the rescue (1/2)

What is Docker?

Docker allows you to package an application with all of its dependencies into a standardized unit for software development*

Containers consist of everything that enables software to run:> Code> Runtime> System Tools > System Libraries

* https://www.docker.com/what-docker

22

Docker Containers to the rescue (2/2)

What can Docker be for SE/WE research?

Docker allows you to package a - Prototype

- Proof-of-concept Implementation - Computational analysis or experiment

with all of its dependencies into a standardized unit for reproducible research

* https://www.docker.com/what-docker

Technical Overview / Virtual Machines vs Containers

“Lightweight” VM > Container is an isolated process (“chroot on steroids”)> Own process space > Own network interface > Feels like a VM > Shares kernel with the host > Isolation through cgroups/namespaces

23

https://www.docker.com/what-is-docker

https://blog.docker.com/2016/03/containers-are-not-vms/

24

Docker Engine

Centralized runtime environment for containers

Enables portability

Sole dependency for Docker

No Emulation layer (almost no performance impact)

https://www.docker.com/products/docker-engine

25

Benefits of Docker Containers (compared to VMs)

Fast instantiation (~1-3 seconds)

Almost native performance

Transparent build process

Smaller Images

Easy to build, share, and publish

* https://www.docker.com/what-docker

25

Benefits of Docker Containers (compared to VMs)

Fast instantiation (~1-3 seconds)

Almost native performance

Transparent build process

Smaller Images

Easy to build, share, and publish

* https://www.docker.com/what-docker

also compared to

other container te

chnologies

26

Single Container Docker Workflow

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container

run

27

Dockerfile Declarative definition of an environment for producing an imageDocker Image Immutable artifact built from a Dockerfile, has one to many layers

Docker ContainerExecution environment - Instantiation/running version of an image (can be parameterized)

Docker MachineCreates Docker hosts on your local Mac or Windows box

Docker RegistryPublic or private repository stores images and allows for their distribution(Docker Hub - https://hub.docker.com/ or CoreOS Quay - https://quay.io/)

Terminology

build

28

Single Container Docker Workflow

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

Image

Docker Image Docker Container

run

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Base Image

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Base ImageBase Image can be an OS (Ubuntu)

or a different, existing image

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base ImageBase Image can be an OS (Ubuntu)

or a different, existing image

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base ImageBase Image can be an OS (Ubuntu)

or a different, existing image

Runs commands as if you were typing them in the command line

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base Image

Install

Base Image can be an OS (Ubuntu) or a different, existing image

Runs commands as if you were typing them in the command line

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base Image

Install

Base Image can be an OS (Ubuntu) or a different, existing image

Runs commands as if you were typing them in the command line

Copies local files from build context into container

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base Image

Install

Volume

Base Image can be an OS (Ubuntu) or a different, existing image

Runs commands as if you were typing them in the command line

Copies local files from build context into container

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base Image

Install

Open PortVolume

Base Image can be an OS (Ubuntu) or a different, existing image

Runs commands as if you were typing them in the command line

Copies local files from build context into container

29

DockerfileDefines the infrastructure and dependencies of a container through instructions

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

COPY redis.conf /var/www/redis.conf

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dependencies

Base Image

Install

Open Port

Start Server

Volume

Base Image can be an OS (Ubuntu) or a different, existing image

Runs commands as if you were typing them in the command line

Copies local files from build context into container

30

Data Volumes

A specially-designated directory within one or more containers that bypasses the Union File System*

Volumes allow you to manage data within containers > Mount a host directory (dependency to the host filesystem)> Mount a data volume container (dependency to another container)> Mount a shared-storage volume (NFS, iSCSI, etc.)

* https://docs.docker.com/engine/userguide/containers/dockervolumes/

31

Single Container Docker Workflow

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container

run

32

Dockerfile —> ImageDefines the infrastructure and dependencies of a container through instructions

docker build -t <imagename> .

32

Dockerfile —> ImageDefines the infrastructure and dependencies of a container through instructions

docker build -t <imagename> .

Build Context containing all local dependencies and

Dockerfile

33

Docker Images

# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB

# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…

33

Docker Images

# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB

# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…

Lists all previously built images

33

Docker Images

# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEmhart/alpine-node latest 2a15d8568f75 1 week ago 36.76 MBhakyll latest d575da1e730c 2 weeks ago 1.487 GBredis alpine 50405530a7e5 4 weeks ago 15.95 MB

# docker rmi hakyllUntagged: hakyll:latestDeleted: sha256:3240943c9ea3f72db51…Deleted: sha256:a3aeefae0d4b8f61…Deleted: sha256:16a7ebd378002f1261…

Lists all previously built images

Removes an image ‘hakyll:latest’

and all its layers from the disk

34

Single Container Docker Workflow

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image

run

Docker Container

35

Image —> Container

docker run -d --name <containername> -p 80:5000 <imagename>

https://docs.docker.com/engine/reference/run/Many more possibilities to run containers, see full reference here:

(A typical example)

35

Image —> Container

docker run -d --name <containername> -p 80:5000 <imagename>

Run container in the background (d for daemon)

https://docs.docker.com/engine/reference/run/Many more possibilities to run containers, see full reference here:

(A typical example)

35

Image —> Container

docker run -d --name <containername> -p 80:5000 <imagename>

Run container in the background (d for daemon)

https://docs.docker.com/engine/reference/run/

Give the container a unique name

Many more possibilities to run containers, see full reference here:

(A typical example)

35

Image —> Container

docker run -d --name <containername> -p 80:5000 <imagename>

Run container in the background (d for daemon)

https://docs.docker.com/engine/reference/run/

Give the container a unique name

Port mappingFirst the exposed port (80)

Second the port within the container (5000)

Many more possibilities to run containers, see full reference here:

(A typical example)

36

Container Managementdocker ps

docker ps -a

docker stop <container>

docker rm <container>

docker rm -v <container>

36

Container Managementdocker psLists all running containers

docker ps -a

docker stop <container>

docker rm <container>

docker rm -v <container>

36

Container Managementdocker psLists all running containers

docker ps -aLists all containers (including the ones that are stopped)

docker stop <container>

docker rm <container>

docker rm -v <container>

36

Container Managementdocker psLists all running containers

docker ps -aLists all containers (including the ones that are stopped)

docker stop <container>Stops a running container

docker rm <container>

docker rm -v <container>

36

Container Managementdocker psLists all running containers

docker ps -aLists all containers (including the ones that are stopped)

docker stop <container>Stops a running container

docker rm <container>Removes a stopped container

docker rm -v <container>

36

Container Managementdocker psLists all running containers

docker ps -aLists all containers (including the ones that are stopped)

docker stop <container>Stops a running container

docker rm <container>Removes a stopped container

docker rm -v <container>Removes a stopped container and its volumes

37

Container Debugging

docker run -ti --entrypoint=bash <imagename>

docker exec -ti <container> bash

docker inspect <container>

docker logs <container>

37

Container Debugging

docker run -ti --entrypoint=bash <imagename>

docker exec -ti <container> bash

docker inspect <container>Low-level information on a container or image

docker logs <container>

37

Container Debugging

docker run -ti --entrypoint=bash <imagename>

docker exec -ti <container> bash

docker inspect <container>Low-level information on a container or image

docker logs <container>Retrieves logs printed to stdout/err within the container

37

Container Debugging

docker run -ti --entrypoint=bash <imagename>

docker exec -ti <container> bashStarts an interactive shell into a running container

docker inspect <container>Low-level information on a container or image

docker logs <container>Retrieves logs printed to stdout/err within the container

37

Container Debugging

docker run -ti --entrypoint=bash <imagename>Starts a container with a different entrypoint

docker exec -ti <container> bashStarts an interactive shell into a running container

docker inspect <container>Low-level information on a container or image

docker logs <container>Retrieves logs printed to stdout/err within the container

38

Docker Hub: Public Registry

39

Pulling Docker Images Getting started with existing images

docker pull nginx:latest

40

Pulling Docker Images Getting started with existing images

docker pull nginx:latestReference to a Docker Image in the Docker Hub

41

Pulling Docker Images Getting started with existing images

docker pull nginx:latestImages can have many “tags”

42

Pulling Docker Images Getting started with existing images

docker pull nginx:latestPulls an image from a Docker registry

43

Pushing Docker Images to a Registry Tag Image

docker tag c6fdd6639541 <username>/<imagename>:<tagname>

Image Id (retrieve through )docker images

44

Pushing Docker Images to a Registry Push Image

44

Pushing Docker Images to a Registry Push Image

docker login --username=<username> --email=<email>

44

Pushing Docker Images to a Registry Push Image

docker login --username=<username> --email=<email>

docker push <username>/<imagename>:<tagname>

45

Case Study 1: ChangeDistiller

https://bitbucket.org/sealuzh/tools-changedistiller

Research Project

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container

run

46

Prerequisites for attending the tutorial

Prerequisites

• git on your terminal

• OSX or Windows: Docker Toolbox, https://www.docker.com/products/docker-toolbox. Then follow the getting started guide on the same page.

• Linux: 1) https://docs.docker.com/linux/, 2) https://docs.docker.com/machine/install-machine/, 3) https://docs.docker.com/compose/install/

Git clone

• git clone https://github.com/ICWE2016DockerTutorial/ChangeDistiller.git

• git clone https://github.com/ICWE2016DockerTutorial/NodeJSApp.git

• git clone https://github.com/ICWE2016DockerTutorial/MongoDB.git

PDF on: http://icwe2016.inf.usi.ch/program/tutorials/docker

47

Case Study 1: ChangeDistiller

https://goo.gl/3h7DGA

We paste the commands we run, live at the following url:

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Base Image

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Copy sources

Base Image

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Copy sources

Base Image

Dependencies

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Copy sources

Base Image

Install

Dependencies

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Copy sources

Base Image

Install

Dependencies

Volume for generated files

48

Case Study 1: ChangeDistiller

(Simple) Dockerfile for ChangeDistiller

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Copy sources

Base Image

Install

Start/enter at analysis

Dependencies

Volume for generated files

49

Case Study 1: ChangeDistiller

docker build -t icwe2016dockertutorial/changedistiller:master .Executes in the folder with the Dockerfile. Creates an image

Illustrative execution of Docker commands

docker run --name cd icwe2016dockertutorial/changedistiller:masterCreates the container and runs the analysis

Context

cd ChangeDistiller

50

Case Study 1: ChangeDistiller Version Pinning

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven=3.0.5-3 RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

50

Case Study 1: ChangeDistiller Version Pinning

FROM java:openjdk-7u101-jdk

ENV app_dir /usr/src/changedistiller

COPY . ${app_dir} WORKDIR ${app_dir}

RUN apt-get update && apt-get install -y maven=3.0.5-3 RUN ${app_dir}/build.sh RUN mvn install

VOLUME ${app_dir}/target

ENTRYPOINT ["mvn", "test"]

Version Pinning for Dependencies

51

Docker Hub: Automated/Trusted Build Improves transparency

https://docs.docker.com/docker-hub/builds/

Docker Hub

51

Docker Hub: Automated/Trusted Build Improves transparency

https://docs.docker.com/docker-hub/builds/

Docker Hubpush

51

Docker Hub: Automated/Trusted Build Improves transparency

https://docs.docker.com/docker-hub/builds/

Docker Hubbuildpush

Recap: Challenges in Reproducibility

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Trusted infrastructure for storage and distribution

52

Recap: Challenges in Reproducibility

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Trusted infrastructure for storage and distribution

Dockerfile

Dockerfile

52

Recap: Challenges in Reproducibility

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Trusted infrastructure for storage and distribution

Dockerfile

Docker Image

Dockerfile

52

Recap: Challenges in Reproducibility

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Trusted infrastructure for storage and distribution

Dockerfile

Docker Image

Docker Container

Dockerfile

52

Recap: Challenges in Reproducibility

> No standard way of describing experiments, environments, (derived) data, and workflows

> No transparency in creating environments and the steps/methods to establish facts or recreate analysis> Experimental nature of research code and ecosystems makes it often hard to build

> Unresolved or undocumented dependencies

> Trusted infrastructure for storage and distribution

Dockerfile

Docker Image

Docker Container

Registries (Docker Hub, Quay, …)

Dockerfile

52

ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

\

Question Time!

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

54

Part III - Multi Containers Docker Application Preview

Reproduce Bugs present in a Version of a CRUD Web Application54

Redis

NodeJS App

MongoDB

network

54

Part III - Multi Containers Docker Application Preview

Reproduce Bugs present in a Version of a CRUD Web Application54

Redis

NodeJS App

link

MongoDB

link

network

54

Part III - Multi Containers Docker Application Preview

Reproduce Bugs present in a Version of a CRUD Web Application54

Redis

NodeJS App

link

MongoDB

link

network

Docker Compose

ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

End of Part II - Lunch Break!

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

Part III - Industrial Case Study

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

57

What is Reproducibility?

57

What is Reproducibility?

Challenges in Reproducibility

57

What is Reproducibility?

Single Container Docker Workflow# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container

run

Challenges in Reproducibility

58

Multi Containers Docker Workflow# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container 1

run

58

Multi Containers Docker Workflow

Docker HubDocker Container 2

run

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container 1

run

network

58

Multi Containers Docker Workflow

Docker HubDocker Container 2

run

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container 1

run

network

58

Multi Containers Docker Workflow

Docker HubDocker Container 2

run

link

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

RUN apt-get install -y gcc make g++ build-essential libc6-dev tcl wget

RUN wget http://download.redis.io/redis-stable.tar.gz -O - | tar -xvz

# RUN tar -zvzf /redis/redis-stable.tar.gz RUN (cd /redis-stable && make) RUN (cd /redis-stable && make test)

RUN mkdir -p /redis-data VOLUME ["/redis-data"] EXPOSE 6379

ENTRYPOINT ["/redis-stable/src/redis-server"] CMD ["--dir", "/redis-data"]

Dockerfile

build

Image

Docker Image Docker Container 1

run

59

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

MongoDB

network

59

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

link

MongoDB

link

network

60

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

60

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

60

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

61

Docker Networks

docker network ls

61

Docker Networks

NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null

docker network ls

61

Docker Networks

docker network create icwe2016Creates a bridge network named icwe2016

NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null

docker network ls

61

Docker Networks

docker network create icwe2016Creates a bridge network named icwe2016

NETWORK ID NAME DRIVERbbd6e3c22dad bridge bridged0bcac7befe4 host hosta4f078c19762 none null

docker network ls

docker network create -d overlay icwe2016Creates an overlay network named icwe2016

62

Docker Networks

docker network create icwe2016

62

Docker Networks

docker network create icwe2016

icwe2016

62

Docker Networks

docker network create icwe2016

icwe2016

docker network connect icwe2016 NodeJSAppConnects NodeJSApp to the icwe2016 network

icwe2016

62

Docker Networks

docker network create icwe2016

icwe2016

docker network connect icwe2016 NodeJSAppConnects NodeJSApp to the icwe2016 network

icwe2016NodeJS App

63

Docker Networks

icwe2016NodeJS App

https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:

63

Docker Networks

icwe2016NodeJS App

docker run -d --net icwe2016 ... MongoDBStarts and connects MongoDB to the icwe2016 network

https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:

63

Docker Networks

icwe2016NodeJS App

docker run -d --net icwe2016 ... MongoDBStarts and connects MongoDB to the icwe2016 network

MongoDB

https://docs.docker.com/engine/reference/commandline/network_create/Many more possibilities for networks, see full reference here:

64

Docker Networks

icwe2016

NodeJS App MongoDB

EXPOSE 27017

64

Docker Networks

icwe2016

NodeJS App MongoDB

mongodb:27017 EXPOSE 27017

link

64

Docker Networks

icwe2016

NodeJS App MongoDB

mongodb:27017 EXPOSE 27017

link

Docker takes care of:

• adding the relevant reference to your container’s /etc/hosts

• keeping the link valid across linked containers’ restarts

65

Case Study 2: NodeJS + MongoDB + Redis

Reproduce Bugs present in a Version of a CRUD Web Application

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

66

Case Study 2: NodeJS + MongoDB + Redis

https://goo.gl/3h7DGA

We paste the commands we run, live at the following url:

67

Case Study 2: Web Application

https://github.com/scotch-io/node-todoRepo with the Application we have started from:

67

Case Study 2: Web Application

https://github.com/scotch-io/node-todoRepo with the Application we have started from:

67

Case Study 2: Web Application

https://github.com/scotch-io/node-todoRepo with the Application we have started from:

67

Case Study 2: Web Application

https://github.com/scotch-io/node-todoRepo with the Application we have started from:

+ a couple of bugs, one client-side and one server-side

68

Case Study 2: NodeJS + MongoDB + Redis

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

69

Case Study 2: NodeJS App Relevant files for Docker

config/database.js

config/redis.js

bin/*

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

Base Image

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

Copy App.

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

Copy App.

Utility

Utility

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

Copy App.

Private Port

Utility

Utility

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

Copy App.

Enter

Private Port

Utility

Utility

70

Case Study 2: NodeJS App(Simple) Dockerfile for NodeJS App

FROM node:6.2.0

RUN mkdir -p /usr/src/app WORKDIR /usr/src/app

COPY package.json /usr/src/app/ RUN npm install

COPY . /usr/src/app

COPY ./bin/wait-for-it.sh / RUN chmod +x /wait-for-it.sh

COPY ./bin/wait-for-mongo-redis.sh / RUN chmod +x /wait-for-mongo-redis.sh

EXPOSE 3000

ENTRYPOINT [ "/wait-for-mongo-redis.sh" ] CMD [ "npm", "start" ]

App. Dir.

Base Image

Install Dependencies

Copy App.

Enter Start

Private Port

Utility

Utility

71

Case Study 2: NodeJS App

docker build -t icwe2016dockertutorial/nodejsapp:master .

Illustrative execution of Docker commands

Context

cd NodeJSApp

72

Case Study 2: NodeJS + MongoDB + Redis

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

73

Case Study 2: MongoDB Relevant files for Docker

data/*

bin/*

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image Defines also the private port

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image

Copy Data

Defines also the private port

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image

Utility

Copy Data

Defines also the private port

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image

Utility

Copy Data

Enter

Defines also the private port

74

Case Study 2: MongoDB(Simple) Dockerfile for MongoDB with Seed data

FROM mongo:3.3.6

COPY data/seed/icwe2016dockertutorial /data/db/icwe2016dockertutorial

COPY bin/initialize-mongo.sh /initialize-mongo.sh RUN chmod +x /initialize-mongo.sh

ENTRYPOINT ["/initialize-mongo.sh"] CMD ["mongod"]

Base Image

Utility

Copy Data

Enter Start

Defines also the private port

75

Case Study 2: MongoDB

docker build -t icwe2016dockertutorial/mongodb:master .

Illustrative execution of Docker commands

Context

cd MongoDB

76

Case Study 2: NodeJS + MongoDB + Redis

Redis

NodeJS App

link

MongoDB

link

network

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Custom Schema and Data

# Build redis from source # Make sure you have the redis source code checked out in # the same directory as this Dockerfile

FROM ubuntu:12.04 MAINTAINER dockerfiles http://dockerfiles.github.io

RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update RUN apt-get upgrade -y

Web App

Illustrative execution of Docker commands

77

Case Study 2: Redis

docker pull redis:3.2.0-alpine

78

Case Study 2: Starting the NodeJS App

docker run -d --net icwe2016 --name mongodb icwe2016dockertutorial/mongodb:masterdocker

docker network create icwe2016

docker run -d --net icwe2016 --name redis redis:3.2.0-alpine

docker run -d --net icwe2016 -p 8080:3000 --name nodejsapp icwe2016dockertutorial/nodejsapp:master

Illustrative execution of Docker commands

79

Case Study 2: Connect to the NodeJs App

79

Case Study 2: Connect to the NodeJs App

79

Case Study 2: Connect to the NodeJs App

docker-machine ls

Illustrative execution of Docker commands

79

Case Study 2: Connect to the NodeJs App

docker-machine ls

Illustrative execution of Docker commands

Better solution (in Beta): https://beta.docker.com

80

Case Study 2: Server-side Fix (live)

docker exec -ti nodejsapp bash

Illustrative execution of commands

Context

app/routes.js

Bug: the App. creates the same item twice

docker restart nodejsapp

apt-get update && apt-get install -y vimvim routers.js \ exit

81

Docker in Continuous Integration and Delivery

Live Fix

CI

CD+

+

81

Docker in Continuous Integration and Delivery

Local Build

ImageLive Fix

CI

CD+

+

81

Docker in Continuous Integration and Delivery

Local Build

ImageLive Fix

CI

CD+

+

Docker Hub Build

Push

81

Docker in Continuous Integration and Delivery

Local Build

ImageLive Fix

CI

CD+

+

Docker Hub Build

Push

81

Docker in Continuous Integration and Delivery

Local Build

ImageLive Fix

CI

CD+

+

Docker Hub Build

82

Case Study 2: Client-side Fix (new build)

docker build -t icwe2016dockertutorial/nodejsapp:master .

Illustrative execution of commands (cd NodeJSApp)

Context

public/js/controllers/main.js

Bug: the App. creates empty todos

docker run -d --net icwe2016 -p 8080:3000 --name nodejsapp icwe2016dockertutorial/nodejsapp:master

docker rm -f -v nodejsapp

83

Case Study 2: Undeploy the App

docker rm -f —v mongodb redis nodejsappIllustrative execution of Docker commands

84

Case Study 2: A Better Way of Starting the NodeJS App

Docker Compose

85

Docker ComposeA tool for defining and running multi-container Docker applications

version: '2'

build:

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

volumes:

85

Docker ComposeA tool for defining and running multi-container Docker applications

version: '2'

build:

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

volumes:

Build settings

85

Docker ComposeA tool for defining and running multi-container Docker applications

version: '2'

build:

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

volumes:

Build settings

Services’ deployment settings

85

Docker ComposeA tool for defining and running multi-container Docker applications

version: '2'

build:

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

volumes:

Build settings

Services’ deployment settings

Networks’ settings

85

Docker ComposeA tool for defining and running multi-container Docker applications

version: '2'

build:

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

volumes:

Build settings

Services’ deployment settings

Networks’ settings

Volumes’ settings

86

Docker ComposeA tool for defining and running multi-container Docker applications

docker-compose up -d

docker-compose start MongoDB

docker-compose logs

https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:

86

Docker ComposeA tool for defining and running multi-container Docker applications

docker-compose up -dStarts the services and detaches from the command

docker-compose start MongoDB

docker-compose logs

https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:

86

Docker ComposeA tool for defining and running multi-container Docker applications

docker-compose up -dStarts the services and detaches from the command

docker-compose start MongoDBStarts the MongoDB service

docker-compose logs

https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:

86

Docker ComposeA tool for defining and running multi-container Docker applications

docker-compose up -dStarts the services and detaches from the command

docker-compose start MongoDBStarts the MongoDB service

docker-compose logsShows the aggregated logs from all the services

https://docs.docker.com/compose/reference/Many more possibilities, see full reference here:

87

Case Study 2: Docker Compose File (1/4)

version: '2'

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

Context

cd NodeJSApp

87

Case Study 2: Docker Compose File (1/4)

version: '2'

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

Services’ deployment settings

Context

cd NodeJSApp

87

Case Study 2: Docker Compose File (1/4)

version: '2'

services: mongodb: ...

redis: ...

nodejsapp: ...

networks: icwe2016: driver: bridge

Services’ deployment settings

Networks’ settings

Context

cd NodeJSApp

88

Case Study 2: Docker Compose File (2/4)

services: mongodb: image: icwe2016dockertutorial/mongodb:master container_name: mongodb # make it explicit the ports exposed by the Dockefile expose: - 27017 # make it explicit the volumes exposed by the Dockerfile volumes: - /data/db - /data/configdb networks: - icwe2016

89

Case Study 2: Docker Compose File (3/4)

redis: image: redis:3.2.0-alpine container_name: redis # make it explicit the ports exposed by the Dockefile expose: - 6379 # make it explicit the volumes exposed by the Dockerfile volumes: - /data networks: - icwe2016

90

Case Study 2: Docker Compose File (4/4)

nodejsapp: image: icwe2016dockertutorial/nodejsapp:master container_name: nodejsapp # make it explicit the ports exposed by the Dockefile expose: - 3000 depends_on: - mongodb - redis ports: - "8080:3000" networks: - icwe2016

91

Case Study 2: Starting the NodeJS App with Docker Compose

docker-compose up -dStarts the services and detaches from the command

docker-compose logs -fShows and follows the aggregated logs from all the services

Illustrative execution of Docker commands

Context

cd NodeJSApp

92

Recap: Multi Containers Deployment

92

Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse

92

Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse

> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version

92

Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse

> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version

> Docker speeds up the debugging of applications composed of many services

92

Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse

> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version

> Docker speeds up the debugging of applications composed of many services

> Docker provides network abstraction for portable deployments of multi container services

92

Recap: Multi Containers Deployment> There are many images on the Docker Hub which we can reuse

> Docker simplifies the reproducibility of bugs because we can always refer to the image built for a specific version

> Docker speeds up the debugging of applications composed of many services

> Docker provides network abstraction for portable deployments of multi container services

> Docker Compose enables portable deployment descriptors

ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

End of Part III - Summary Time!

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

94

Steps for ReproducibilityA recipe for enhancing reproducibility with Docker

94

Steps for ReproducibilityA recipe for enhancing reproducibility with Docker

> Provide the source code (documented) on a public repository

94

Steps for ReproducibilityA recipe for enhancing reproducibility with Docker

> Provide the source code (documented) on a public repository

> Release the Dockerfiles using version pinning

94

Steps for ReproducibilityA recipe for enhancing reproducibility with Docker

> Provide the source code (documented) on a public repository

> Release the Dockerfiles using version pinning

> Setup an automated/trusted build on the Docker Hub

94

Steps for ReproducibilityA recipe for enhancing reproducibility with Docker

> Provide the source code (documented) on a public repository

> Release the Dockerfiles using version pinning

> Setup an automated/trusted build on the Docker Hub

> Provide portable deployment descriptors using Docker Compose

95

Steps for ReproducibilityAdditional useful services

> DOI for Source Code and Docker Images:

• Citable Codehttps://guides.github.com/activities/citable-code/

95

Steps for ReproducibilityAdditional useful services

> DOI for Source Code and Docker Images:

• Citable Codehttps://guides.github.com/activities/citable-code/

• Citable Docker Images http://www.software.ac.uk/blog/2016-03-29-reproducible-research-citing-your-execution-environment-using-docker-and-doi

96

Docker’s Limitations

96

Docker’s Limitations

> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]

96

Docker’s Limitations

> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]

> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]

96

Docker’s Limitations

> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]

> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]

> Limits with proprietary software and dependencies

96

Docker’s Limitations

> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]

> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]

> Limits with proprietary software and dependencies

> Non-Disclosure Agreements / Intellectual Property

96

Docker’s Limitations

> Not enough for performance experiments [Jimenez et al., The Role of Container Technology in Reproducible Computer Systems Research]

> Not enough for data-intensive workload[http://ivory.idyll.org/blog/2015-docker-and-replicating-papers.html, data on http://www.zenodo.org]

> Limits with proprietary software and dependencies

> Non-Disclosure Agreements / Intellectual Property

> Can we build the same artifact from the specification (Dockerfile) even in 10 years? [Suggestion: Version Pinning]

Sum up and Conclusion

> Containers enable a standard, fast, and easy way of describing experiments and environments

> Containers help your future self, reviewers, and other researchers and colleagues to make use and improve your work

98

Further Reading> Docker for beginners: http://prakhar.me/docker-curriculum/

> Docker Hands-on: https://github.com/alexellis/HandsOnDocker/blob/master/Labs.md

> Experience in using Docker for reproducible research: http://melissagymrek.com/science/2014/08/29/docker-reproducible-research.html

> Multi-host networking with Docker: https://docs.docker.com/engine/userguide/networking/get-started-overlay/

ICWE 2016, Lugano

Photo Credits: Astrid Westvang, https://flic.kr/p/pWJLCW

Question Time!

Jürgen Cito, Vincenzo Ferme, Harald C. Gall

Material: https://github.com/ICWE2016DockerTutorial https://hub.docker.com/u/icwe2016dockertutorial/

Speakers References + Slides + Material

Slides: http://icwe2016.inf.usi.ch/program/tutorials/docker

Vincenzo Ferme PhD @ USIfermev@usi.ch@VincenzoFerme

Jürgen Cito PhD @ UZH

cito@ifi.uzh.ch@citostyle