Docker and Maestro for fun, development and profit
-
Upload
maxime-petazzoni -
Category
Technology
-
view
4.264 -
download
1
description
Transcript of Docker and Maestro for fun, development and profit
![Page 1: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/1.jpg)
Docker and MaestroFor fun, development and profit
![Page 3: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/3.jpg)
Real-time monitoring, instrumentation, observability and analytics
Still in “stealth” mode Get updates at www.signalfuse.com
![Page 4: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/4.jpg)
–You, some time in the last hour (hopefully).
“Docker is awesome!”
![Page 5: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/5.jpg)
A versatile foundationService or application containment, security, software delivery, host
and environment isolation, …and so much more.
![Page 6: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/6.jpg)
Power at your fingertipsComplete control through the remote API
Available programmatic clients like docker-py
![Page 7: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/7.jpg)
docker:$ docker -d -H tcp://0.0.0.0:4243 !client:$ cat << EOF | python - import docker from pprint import pprint as pp pp(docker.client.Client(‘tcp://docker:4243') .images('quay.io/signalfuse/maestro-base')) EOF !![{u’Created': 1391202535, u’Id': u’37de13d273eb9a02cd64…’, u’Repository': u'quay.io/signalfuse/maestro-base', u'Size': 155663843, u'Tag': u'0.1.6', u'VirtualSize': 774767942}]
![Page 8: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/8.jpg)
Docker’s Achilles: orchestration
Single-host is alright with links, but multi-host just isn’t there.
![Page 9: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/9.jpg)
How do I orchestrate the deployment and control of a full, multi-host,
Docker-based environment?
![Page 10: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/10.jpg)
How do I make this process one and the same for development, testing
and production environments?
(And more importantly:)
![Page 11: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/11.jpg)
Enter: MaestroThe totally not scalable, pet project that solved my use case.
(and maybe yours)
![Page 12: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/12.jpg)
Maestro is actually MaestroNG, a re-invention of Kimbro Staken’s Maestro
(formerly, dockermix)
![Page 13: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/13.jpg)
Takes in a definition of services, their dependencies , configuration and target host…
!
…and automates the deployment (and control) of their corresponding containers on these hosts.
![Page 14: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/14.jpg)
Classic use case: a pool of “dumb” workers on your favorite cloud/hosting provider that just run Docker.
!
No need to (ma)ssh into anything, no need to pre-configure anything.
!
Everything is remote controlled.
![Page 15: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/15.jpg)
Other typical use case: running all the components of your stack in a single, local virtual machine.
!
Useful for development, integration testing, etc.
![Page 16: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/16.jpg)
Philosophy: lightweight application/service containers. !
Represent and control your software stack and its dependencies.
!
Docker images are the output of your CI process (automation!).
!
Start fast, fail faster. Not for heavyweight, complex container “VMs”.
![Page 17: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/17.jpg)
Each service instance (container) defines where it runs and which ports it exposes, among other things.
!
Like Docker links, Maestro works by injecting this information in the container’s environment about each
container’s service’s dependencies.
![Page 18: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/18.jpg)
MAESTRO_ENVIRONMENT_NAME = lspe SERVICE_NAME = kafka CONTAINER_NAME = broker-1 CONTAINER_HOST_ADDRESS = 192.168.10.2 !ZOOKEEPER_ZK_NODE_1_HOST = 192.168.10.2 ZOOKEEPER_ZK_NODE_1_CLIENT_PORT = 2181 ZOOKEEPER_ZK_NODE_1_PEER_PORT = 2888 ZOOKEEPER_ZK_NODE_1_LEADER_ELECTION_PORT = 3888 !KAFKA_BROKER_1_HOST = 192.168.10.2 KAFKA_BROKER_1_BROKER_INTERNAL_PORT = 9042 KAFKA_BROKER_1_BROKER_PORT = 9042 KAFKA_BROKER_1_JMX_INTERNAL_PORT = 7199 KAFKA_BROKER_1_JMX_PORT = 17199
Let’s say broker-1 of kafka depends on ZooKeeper. Its environment will contain:
![Page 19: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/19.jpg)
<SERVICE_NAME>_<CONTAINER_NAME>_HOST <SERVICE_NAME>_<CONTAINER_NAME>_PORT <SERVICE_NAME>_<CONTAINER_NAME>_INTERNAL_PORT
![Page 20: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/20.jpg)
Using this information, you can configure your application at container start time.
!
If you like Python, Maestro helps you by providing a set of guest helper functions in maestro.guestutils to easily
extract and use this data.
![Page 21: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/21.jpg)
#!/usr/bin/env python !# This is my cool container’s “init script” !import os from maestro.guestutils import * !os.execl(‘java’, ‘java’, ‘-jar’, ‘my-app.jar’, ‘-DlistenPort={}’.format(get_port(‘service’)), ‘-DzkServers={}’.format( get_node_list(‘zookeeper’, ports=[‘peer’])))
![Page 22: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/22.jpg)
Dependency order is respected on start; inverse order on stop.
!
Can be overridden to stop individual services or containers.
![Page 23: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/23.jpg)
MyApp Kafka ZK
Start order: 1. ZooKeeper 2. Kafka 3. MyApp
Stop order: 1. MyApp 2. Kafka 3. ZooKeeper
Works on subsets of services too.
![Page 24: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/24.jpg)
So how do you wield this power?
A bit clunkily, with YAML (and a bit of Jinja2). !!!
(sorry)
![Page 25: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/25.jpg)
# Yay, YAML! name: lspe !registries: # Define custom image registries for # private registries, with credentials. !ships: # Declare each target host. # (Docker daemon locations) !services: # Declare each service, their # instances, dependencies and # configuration
![Page 26: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/26.jpg)
registries: # Quay.io with Maestro robot account quay.io: registry: https://quay.io/v1/ email: [email protected] username: signalfuse+maestro password: {{ env.SUPER_SECRET }}
When starting a container, Maestro will automatically login and pull the image from the right place if the image
name matches a configured registry.
![Page 27: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/27.jpg)
ships: # Local virtual machine vm: ip: 192.168.10.2 docker_port: 4243 timeout: 10 # Slow VM is slow # A shorter form… vm2: {ip: 192.168.10.3, timeout: 5}
Ships carry containers and are referred to by name in the configuration.
![Page 28: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/28.jpg)
services: # ZooKeeper zookeeper: image: quay.io/signalfuse/zookeeper:3.4.5 ! # Our zoo isn’t too wild, # only one keeper is enough. zk-node-1: ship: vm ports: client: 2181 peer: 2888/tcp leader_election: “3888/tcp:3888/tcp” # Keep persistent data on the host. volumes: /var/lib/zookeeper: /data/zookeeper # Environment can be passed-in too. env: JVM_FLAGS: “-Xmx1g”
![Page 29: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/29.jpg)
# Kafka kafka: image: quay.io/signalfuse/kafka:0.8.0 requires: [ zookeeper ] env: ZOOKEEPER_BASE: /lspe/kafka RETENTION_HOURS: 48 broker-1: ship: vm ports: {broker: 9092, jmx: “7199:17199”} # Keep persistent data on the host. volumes: /var/lib/kafka: /data/kafka env: BROKER_ID: 0
More flexibility in port mappings, volume bindings, and environment variables definition not shown here.
![Page 30: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/30.jpg)
See README.md for full syntax details and featureshttps://github.com/signalfuse/maestro-ng/blob/master/README.md
![Page 31: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/31.jpg)
Demo time!Be prepared for it to fail, because demos always do.
![Page 32: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/32.jpg)
What’s next?More flexible service status detection (not only port pinging)
Soft and hard service dependencies Parallel startup of independent services and instances of a service
![Page 33: Docker and Maestro for fun, development and profit](https://reader033.fdocuments.us/reader033/viewer/2022060108/55515220b4c905f2288b5602/html5/thumbnails/33.jpg)
That’s it!Thanks for listening! :)
github.com/dotcloud/docker-py github.com/signalfuse/maestro-ng