D o c k e r - i n s ta l l a ti o n a n d fi r s t s te p...

20.03.2020 2_docker_intro

file:///home/szwabin/Dropbox/Praca/Zajecia/BigData/Lectures/2_docker_intro.html 1/16

Docker - installation and first steps

JS, 5.03.2019Outlook:

What is Docker?Docker architectureInstallation

Linux (Ubuntu)WindowsMacOS

First stepsDockerize your Python appUsing Hadoop with Docker

What is Docker?https://www.docker.com/ (https://www.docker.com/)a platform for developers and sysadmins to develop, ship, and run applicationscomponents:

Docker Engine - open source containerization technology combined with a work flow for building andcontainerizing applicationsDocker Hub - a SaaS service for sharing and managing application stacks

Docker allows to separate applications from infrastructure (quick delivery of software)Docker provides the ability to package and run an application in a loosely isolated environment called acontainermany containers can simultaneously run on a given hostcontainers are lightweight:

they do not need the extra load of a hypervisorthey run directly within the host machine’s kernelyou can run more containers on a given hardware combination than if you were using virtual machinesyou can run Docker containers within host machines that are actually virtual machines

https://www.docker.com/



Docker Architecturea client-server architecture

on the same machine via UNIX socketsvia network interface on remote machines

Docker deamon

listens for Docker API requestsmanages Docker objects such as images, containers, networks, and volumesa daemon can also communicate with other daemons to manage Docker services

Docker client

the primary way to interact with Dockercommunicates with a deamon via the Docker REST API

Docker registries

a Docker registry stores Docker imagesDocker Hub (https://hub.docker.com/ (https://hub.docker.com/)) is a public registry that anyone can use

Docker is configured to look for images on Docker Hub by defaulta user can run his own private registryif Docker Datacenter (DDC) is used, it includes Docker Trusted Registry (DTR)

Docker objects

Images

a read-only template with instructions for creating a Docker containeroften, an image is based on another image, with some additional customization

for example: an image based on the ubuntu image, but with the Apache web server and user'sapplication, as well as the configuration details needed to make the application run

a user may create his own images or only use those created by others and published in a registry

https://hub.docker.com/



Containers

a runnable instance of an imagea user can create, start, stop, move, or delete a container using the Docker API or CLIa container may be connected to one or more networksstorage may be attached to a containera new image may be created based on the current state of a containerby default, a container is relatively well isolated from other containers and its host machinewhen a container is removed, any changes to its state that are not stored in persistent storage disappear

Services

allow to scale containers across multiple Docker daemons (all work together as a swarm with multiple managersand workers)by default, the service is load-balanced across all worker nodesto the consumer, the Docker service appears to be a single applicationsupported in Docker 1.12 and higher

InstallationDocker Community Edition (CE) is ideal for developers and small teams looking to get started with Docker

Linux (Ubuntu)

Remove unnecessary Docker versions (if required):

sudo apt-get purge docker lxc-docker docker-engine docker.io

Install some required packages:

sudo apt-get install curl apt-transport-https ca-certificates software-properties

-common

Import docker's official GPG key to verify packages signature before installing them with apt-get:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add

Add the Docker repository on your Ubuntu system which contains Docker packages including its dependencies. You musthave to enable this repository to install Docker on Ubuntu:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu

$(lsb_release -cs) stable"

Install the software:

sudo apt-get update

sudo apt-get install docker-ce

You can verify the status of the service with

sudo systemctl status docker



MS Windows

system requirementsWindows 10 64bit: Pro, Enterprise or Education (1607 Anniversary Update, Build 14393 or later)Virtualization enabled in BIOSSLAT-capable CPUat least 4GB of RAMif your system does not meet those requirements, try Docker Toolbox(https://docs.docker.com/toolbox/overview/ (https://docs.docker.com/toolbox/overview/))

download Docker Desktop for Windows from Docker Hub (https://hub.docker.com/editions/community/docker-ce-desktop-windows (https://hub.docker.com/editions/community/docker-ce-desktop-windows))important note

Docker Desktop for Windows requires Microsoft Hyper-V to runthe installer will enable Hyper-V for you, if needed, and will restart your machineafter Hyper-V is enabled, VirtualBox no longer works, but any VirtualBox VM images remain

double-click Docker Desktop for Windows Installer.exe to run the installerfollow the install wizard to accept the license, authorize the installer, and proceed with the install

you will be asked to authorize Docker.app with your system password during the install processprivileged access is needed to install networking components, links to the Docker apps, and managethe Hyper-V VMs

click Finish on the setup complete dialog to launch Docker.

Mac OS

system requirementsMac hardware must be a 2010 or newer model, with Intel’s hardware support for memory managementunit (MMU) virtualization, including Extended Page Tables (EPT) and Unrestricted ModemacOS Sierra 10.12 and newer macOS releases are supportedat least 4GB of RAMVirtualBox prior to version 4.3.30 must NOT be installed (it is incompatible with Docker Desktop forMac)if your system does not meet those requirements, try Docker Toolbox(https://docs.docker.com/toolbox/overview/ (https://docs.docker.com/toolbox/overview/))

download Docker Desktop for Mac from Docker Hubdouble-click Docker.dmg to open the installer, then drag Moby the whale to the Applications folder

double-click Docker.app in the Applications folder to start Docker

https://docs.docker.com/toolbox/overview/

https://hub.docker.com/editions/community/docker-ce-desktop-windows

https://docs.docker.com/toolbox/overview/



First stepsLet us run our first container:

In [1]: !docker run ubuntu /bin/echo 'Hello world'

The Docker daemon binds to a Unix socket, which by default is owned by the user root and other users can only accessit using sudo:

szwabin@enterprise:~$ sudo docker run ubuntu /bin/echo 'Hello world'

[sudo] hasło użytkownika szwabin:

Unable to find image 'ubuntu:latest' locally

latest: Pulling from library/ubuntu

6cf436f81810: Pull complete

987088a85b96: Pull complete

b4624b3efe06: Pull complete

d42beb8ded59: Pull complete

Digest: sha256:7a47ccc3bbe8a451b500d2b53104868b46d60ee8f5b35a24b41a86077c650210

Status: Downloaded newer image for ubuntu:latest

Hello world

If we dont want to preface the docker command with sudo, we have to create a group called docker and add users toit:

sudo groupadd docker

sudo usermod -aG docker $USER

Then we should logout and login to get the new group permissions:

In [1]: !docker run ubuntu /bin/echo 'Hello world'

The ubuntu image is one of the Docker Official Images that may be found on Docker Hub.

Another one "Hello world" example:

docker: Got permission denied while trying to connect to the Docker daemon s

ocket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.soc

k/v1.39/containers/create: dial unix /var/run/docker.sock: connect: permissi

on denied.

See 'docker run --help'.

Hello world



In [2]: !docker run hello-world

Searching for images in the registry is possible:

Unable to find image 'hello-world:latest' locally

latest: Pulling from library/hello-world

1b930d010525: Pull complete 977B/977BBDigest: sha256:2557e3c07ed1e38f26e389

462d03ed943586f744621577a99efb77324b0fe535

Status: Downloaded newer image for hello-world:latest

Hello from Docker!

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.

2. The Docker daemon pulled the "hello-world" image from the Docker Hub.

(amd64)

3. The Docker daemon created a new container from that image which runs the

executable that produces the output you are currently reading.

4. The Docker daemon streamed that output to the Docker client, which sent

it

to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:

https://hub.docker.com/

For more examples and ideas, visit:

https://docs.docker.com/get-started/



In [3]: !docker search python

In the OFFICIAL column, OK indicates an image built and supported by the company behind the project.

Once we have identified the image that we would like to use, we can download it to our computer:

NAME DESCRIPTION

STARS OFFICIAL AUTOMATED

python Python is an interpreted, interactiv

e, objec… 3947 [OK]

django Django is a free web application fram

ework, … 798 [OK]

pypy PyPy is a fast, compliant alternative

implem… 171 [OK]

kaggle/python Docker image for Python scripts run o

n Kaggle 109 [OK]

frolvlad/alpine-python3 The smallest Docker image with Python

3.5 (~… 96 [OK]

azuresdk/azure-cli-python For official Azure CLI 2.0 builds, se

e https… 37 [OK]

centos/python-35-centos7 Platform for building and running Pyt

hon 3.5… 33

arm32v7/python Python is an interpreted, interactiv

e, objec… 33

circleci/python Python is an interpreted, interactiv

e, objec… 27

joyzoursky/python-chromedriver Python with Chromedriver, for running

automa… 24 [OK]


hon 3.6… 16


hon 2.7… 13

iron/python Tiny Python Microcontainer

9

civisanalytics/datascience-python Common dependencies for data science

workflo… 8 [OK]

microsoft/azure-functions-python3.6 Azure functions python image

8 [OK]

publicisworldwide/python-conda Basic Python environments with Conda.

6 [OK]

dockershelf/python Repository for docker images of Pytho

n. Test… 4 [OK]

civisanalytics/civis-jupyter-python3 Custom Jupyter Notebook image for Pyt

hon 3 3 [OK]

bitnami/python Bitnami Python Docker Image

3 [OK]

komand/python-plugin DEPRECATED: Komand Python SDK

2 [OK]


hon 3.4… 2

civisanalytics/civis-jupyter-python2 Custom Jupyter Notebook image for Pyt

hon 2. 1 [OK]

saagie/python Repo for python jobs

0

qbtrade/python python 3.5.0 with requirements

0

openshift/python-33-centos7 DEPRECATED: A Centos7 based Python v

3.3 imag… 0



In [4]: !docker pull python

To see the images that have been downloaded to your computer, type:

In [5]: !docker images

In the above examples, the containers finished the job just after emitting the text message. An iterative access to thecontainers is possible with the -it flag.

docker run -it python

Python 3.7.2 (default, Mar 5 2019, 06:22:51)

[GCC 6.3.0 20170516] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>>

docker run -it ubuntu

root@38b593981bdc:/#

We can view active containers with

In [6]: !docker ps

To stop a container, simply issue the command

In [8]: !docker stop 38b593981bdc

Using default tag: latest

latest: Pulling from library/python

22dbe790f715: Pulling fs layer

0250231711a0: Pulling fs layer

6fba9447437b: Pulling fs layer

c2b4d327b352: Pulling fs layer

270e1baa5299: Pulling fs layer

8dc8edf0ab44: Pulling fs layer

86ded05de41b: Pulling fs layer

1eac5266a8fa: Pulling fs layer

61b3f1392c29: Pull complete 813MB/1.813MBBKDigest: sha256:1b9f945627c20879af

ab20f5da7607c6591959c053c7075bd8e9913d4d6edf73

Status: Downloaded newer image for python:latest

REPOSITORY TAG IMAGE ID CREATED

SIZE

python latest 32260605cf7a 6 hours ago

929MB

ubuntu latest 47b19964fb50 3 weeks ago

88.1MB

hello-world latest fce289e99eb9 2 months ago

1.84kB

CONTAINER ID IMAGE COMMAND CREATED

STATUS PORTS NAMES

38b593981bdc ubuntu "/bin/bash" About a minute a

go Up About a minute nifty_ptolemy

38b593981bdc



In [9]: !docker ps

Let us have a look at other functionalities of the docker client:

CONTAINER ID IMAGE COMMAND CREATED

STATUS PORTS NAMES



In [10]: !docker



Usage: docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Options:

--config string Location of client config files (default

"/home/szwabin/.docker")

-D, --debug Enable debug mode

-H, --host list Daemon socket(s) to connect to

-l, --log-level string Set the logging level

("debug"|"info"|"warn"|"error"|"fatal")

(default "info")

--tls Use TLS; implied by --tlsverify

--tlscacert string Trust certs signed only by this CA (default

"/home/szwabin/.docker/ca.pem")

--tlscert string Path to TLS certificate file (default

"/home/szwabin/.docker/cert.pem")

--tlskey string Path to TLS key file (default

"/home/szwabin/.docker/key.pem")

--tlsverify Use TLS and verify the remote

-v, --version Print version information and quit

Management Commands:

builder Manage builds

config Manage Docker configs

container Manage containers

engine Manage the docker engine

image Manage images

network Manage networks

node Manage Swarm nodes

plugin Manage plugins

secret Manage Docker secrets

service Manage services

stack Manage Docker stacks

swarm Manage Swarm

system Manage Docker

trust Manage trust on Docker images

volume Manage volumes

Commands:

attach Attach local standard input, output, and error streams to a ru

nning container

build Build an image from a Dockerfile

commit Create a new image from a container's changes

cp Copy files/folders between a container and the local filesyste

m

create Create a new container

diff Inspect changes to files or directories on a container's files

ystem

events Get real time events from the server

exec Run a command in a running container

export Export a container's filesystem as a tar archive

history Show the history of an image

images List images

import Import the contents from a tarball to create a filesystem imag

e

info Display system-wide information

inspect Return low-level information on Docker objects

kill Kill one or more running containers

load Load an image from a tar archive or STDIN

login Log in to a Docker registry

logout Log out from a Docker registry

logs Fetch the logs of a container

pause Pause all processes within one or more containers

port List port mappings or a specific mapping for the container

ps List containers

pull Pull an image or a repository from a registry

push Push an image or a repository to a registry



Dockerize your Python appConsider a simple "Hello world" application written in flask:

In [ ]: # %load flask_example/index.py

from flask import Flask

app = Flask(__name__)

@app.route("/")

def hello():

return "Hello World (from a container)!"

if __name__ == "__main__":

app.run(host="0.0.0.0", port=int("5000"), debug=True)

Our goal is to deploy the code via a Docker image. To this end we have to create a text file called Dockerfile in the rootdirectory of our application:

In [ ]: # %load flask_example/Dockerfile

FROM python:alpine3.7

COPY . /app

WORKDIR /app

RUN pip install -r requirements.txt

EXPOSE 5000

CMD ["python","/app/index.py","-p 5000"]

FROM is telling Docker what base image to use for the container (Python 3.7 installed on Alpine Linux, aminimalist Linux distro)COPY moves the application into the container imageWORKDIR sets the working directoryEXPOSE exposes a port used by FlaskCMD tells the container what to execute to start the applicationRUN is calling PyPi (pip) and pointing to the requirements.txt file, which contains a list of the dependencies (flaskin this case)

In [ ]: # %load flask_example/requirements.txt

flask

rename Rename a container

restart Restart one or more containers

rm Remove one or more containers

rmi Remove one or more images

run Run a command in a new container

save Save one or more images to a tar archive (streamed to STDOUT b

y default)

search Search the Docker Hub for images

start Start one or more stopped containers

stats Display a live stream of container(s) resource usage statistic

s

stop Stop one or more running containers

tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

top Display the running processes of a container

unpause Unpause all processes within one or more containers

update Update configuration of one or more containers

version Show the Docker version information

wait Block until one or more containers stop, then print their exit

codes

Run 'docker COMMAND --help' for more information on a command.



To build the image, run Docker build from a command line or terminal that is in the root directory of the application:

docker build --tag my-python-app .

You should see the following output in the terminal:



Sending build context to Docker daemon 4.096kB

Step 1/6 : FROM python:alpine3.7

alpine3.7: Pulling from library/python

48ecbb6b270e: Pull complete

692f29ee68fa: Pull complete

6439819450d1: Pull complete

3c7be240f7bf: Pull complete

ca4b349df8ed: Pull complete

Digest: sha256:35f6f83ab08f98c727dbefd53738e3b3174a48b4571ccb1910bae480dcdba847

Status: Downloaded newer image for python:alpine3.7

---> 00be2573e9f7

Step 2/6 : COPY . /app

---> cb030c04d394

Step 3/6 : WORKDIR /app

---> Running in 20e2dbec2ae8

Removing intermediate container 20e2dbec2ae8

---> 80e727875a00

Step 4/6 : RUN pip install -r requirements.txt

---> Running in 7e187bba9c50

Collecting flask (from -r requirements.txt (line 1))

Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14d

acb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)

Collecting Werkzeug>=0.14 (from flask->-r requirements.txt (line 1))

Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c47

64e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)

Collecting Jinja2>=2.10 (from flask->-r requirements.txt (line 1))

Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7be

d8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)

Collecting click>=5.1 (from flask->-r requirements.txt (line 1))

Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c

434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)

Collecting itsdangerous>=0.24 (from flask->-r requirements.txt (line 1))

Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c2

4d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl

Collecting MarkupSafe>=0.23 (from Jinja2>=2.10->flask->-r requirements.txt (line

1))

Downloading https://files.pythonhosted.org/packages/b9/2e/64db92e53b86efccfaea713

21f597fa2e1b2bd3853d8ce658568f7a13094/MarkupSafe-1.1.1.tar.gz

Building wheels for collected packages: MarkupSafe

Building wheel for MarkupSafe (setup.py): started

Building wheel for MarkupSafe (setup.py): finished with status 'done'

Stored in directory: /root/.cache/pip/wheels/f2/aa/04/0edf07a1b8a5f5f1aed7580fffb

69ce8972edc16a505916a77

Successfully built MarkupSafe

Installing collected packages: Werkzeug, MarkupSafe, Jinja2, click, itsdangerous, f

lask

Successfully installed Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 click-7.0 flask

-1.0.2 itsdangerous-1.1.0

Removing intermediate container 7e187bba9c50

---> 67d56cc4c274

Step 5/6 : EXPOSE 5000

---> Running in 8d2f8d220ba0

Removing intermediate container 8d2f8d220ba0

---> b2fe0fc9f86a

Step 6/6 : CMD python ./index.py

---> Running in 88485e814d3b

Removing intermediate container 88485e814d3b

---> afc48c9405fe



Successfully built afc48c9405fe

Successfully tagged my-python-app:latest

We can run the app with:

docker run --name python-app -p 5000:5000 my-python-app

Here:

--name - name of the container-p - a mapping between the host’s port 5000 to the containers port of 5000my-python-app - image to run

Using Hadoop with Dockerclodera/quickstart is a single-node deployment of Cloudera's 100% open-source Hadoop platform, andCloudera Manager

In [5]: !docker pull cloudera/quickstart:latest

In [6]: !docker images

latest: Pulling from cloudera/quickstart

1d00652ce734: Pull complete 444GB/4.444GBBKDownloading 10.2MB/4.444GBDownl

oading 595.5MB/4.444GB1d00652ce734: Downloading 2.264GB/4.444GBExtracting

3.326GB/4.444GBDigest: sha256:f91bee4cdfa2c92ea3652929a22f729d4d13fc838b00f1

20e630f91c941acb63

Status: Downloaded newer image for cloudera/quickstart:latest

REPOSITORY TAG IMAGE ID CREATED

SIZE

my-python-app latest 737848ec50ca 18 hours ago

91.7MB

<none> <none> 8aa7cbd60bb9 18 hours ago

91.7MB

<none> <none> 19cff0b28434 18 hours ago

91.7MB

<none> <none> 1d0f16ae7373 18 hours ago

91.7MB

<none> <none> 87ad11cac993 18 hours ago

91.7MB

<none> <none> a9ecdd9d7c6d 18 hours ago

91.7MB

<none> <none> afc48c9405fe 18 hours ago

91.7MB

python latest 32260605cf7a 25 hours ago

929MB

ubuntu latest 47b19964fb50 4 weeks ago

88.1MB

python alpine3.7 00be2573e9f7 4 weeks ago

81.3MB

hello-world latest fce289e99eb9 2 months ago

1.84kB

cloudera/quickstart latest 4239cd2958c6 2 years ago

6.34GB



We can run the image with

docker run --hostname=quickstart.cloudera --privileged=true -t -i [OPTIONS] [IMAGE]

/usr/bin/docker-quickstart

The meaning of the required flags and other options is the following:

--hostname=quickstart.cloudera Required: pseudo-distributed configuration assumes this hostname--privileged=true Required: for HBase, MySQL-backed Hive metastore, Hue, Oozie, Sentry, and ClouderaManager, and possibly others-t Required: once services are started, a Bash shell takes over and will die without this-i Required: if you want to use the terminal, either immediately or attach later-p 8888 Recommended: maps the Hue port in the guest to another port on the host-p [PORT] Optional: map any other ports (e.g. 7180 for Cloudera Manager, 80 for a guided tutorial)-d Optional: runs the container in the background

In [ ]:

D o c k e r - i n s ta l l a ti o n a n d fi r s t s te p...

Documents

Transcript of D o c k e r - i n s ta l l a ti o n a n d fi r s t s te p...