From Lisp to Clojure/Incanter and R - Find Meetup groups near you
R meetup 20161011v2
-
Upload
niels-ole-dam -
Category
Data & Analytics
-
view
55 -
download
2
Transcript of R meetup 20161011v2
28/1-14 23.24Things in Flow | Helping teams and individuals to get things done in flow
Page 1 of 2http://thingsinflow.dk/
Search …
Archives
January 2014
Meta
Log in
Things in Flow
THINGS IN FLOW APS · PARKVEJ 3 · 2830 VIRUM · CVR 35632506 · EMAIL: [email protected] · THEME: MOTIF BY AUTOMATTIC.
HOME SAMPLE PAGE
1 COMMENT
Hello world!
12/01/2014
Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!
How to autodeploy your R solution to DigitalOcean using
Docker, GitHub and CircleCI
CopenhagenR - useR Group Meetup IT University of Copenhagen, Oct. 11 2016Presenter: Niels Ole Dam, Things in Flow
28/1-14 23.24Things in Flow | Helping teams and individuals to get things done in flow
Page 1 of 2http://thingsinflow.dk/
Search …
Archives
January 2014
Meta
Log in
Things in Flow
THINGS IN FLOW APS · PARKVEJ 3 · 2830 VIRUM · CVR 35632506 · EMAIL: [email protected] · THEME: MOTIF BY AUTOMATTIC.
HOME SAMPLE PAGE
1 COMMENT
Hello world!
12/01/2014
Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!
Things in Flow
About me
• Physics and Communications at Roskilde University
• Previously: • Project Manager at Experimentarium science center • Science Journalist at Illustrated Science magazine • Digital Business Developer at Bonnier Publications • Project Manager and Drupal Teamlead at Peytz & Co • Games Producer at Cape Cph
• Since 2014: • Independent consultant at Things in Flow
• I collect tools and help people get their stuff done digitally and with the right amount of efficient processes so that the work still feels right
2
Things in Flow
Coursera courses to recommend
• Data Science Specialisation by Johns Hopkins University • I did 8 of them a couple of years ago – they where great.
• Process Mining by Eindhoven University of Technology • I did this one too – and I’m actually an official Disco partner.
• By The Way: • I’m currently looking for a good Process Mining case. • Please contact me if you have that kind of log file data and
business itch that needs scratching.
3
Tip: All underlined links are clickable in the pdf-version of this slideshow – click to get more info on the subjects.
Things in Flow
Why I chose R
• Feel the raw data under my fingertips (again)
• Visualie raw data
• Deliver prototypes to colleagues/clients
• Deliver widely available solutions to clients
• Bonus: Test new tools without polluting my computer
4
Things in Flow
Why I chose Docker
• Production ready for teams and small companies
• Makes it possible to follow the Continuous Integration (CI) paradigme in my own production
• Open Source – dedication and low cost
• Docker with R is Awesome! :-)
5
Hang on! A lot of steps and things to grasp but easy to execute in practice
;-)
Things in Flow
Ingredients to shop for• GitHub account:
• Free if Open Source – otherwise starting at $7/mo
• Docker Hub account (optional): • Free if Open Source and only 1 private account, otherwise starting at $7/mo
• CircleCI: • Free if Linux and only 1 container – otherwise starting at $39-50/mo
• R + RStudio locally: • Free
• Cloud Server: • Variable, typically starting at $5/mo
• Total: • Starting with Free (Open Source) or $7/m (1 private project) + Cloud server
7
GitHub +
RStudio
Things in Flow
GitHub: How it works
• Clone code from GitHub
• Commit your own changes
• Git keeps track on additions and removals automatically
• Push your code back to GitHub
9
Source: betterexplained.com
Things in Flow
Using Git with RStudio
• Create a repo at GitHub
• Clone the repo to a local RStudio project
• Commit to GitHub regularly using RStudio
Use branching for trying out alternative solutions: • Open a terminal via the RStudio-link and run:
git checkout -b [branchname] • …do your stuff with local commits… • Switch to Master, open a terminal via the RStudio-link and run:
git merge --no-ff [branchname] • …the branch is now merged back into master with all it’s Git history…
10
Things in Flow
Demo 1: GitHub + RStudio
• Connect RStudio project with GitHub
• Make a RStudio Shiny App project
• Make some code changes
• Push changes to GitHub repo
• Optional: Make a local branch
11
Things in Flow
New GitHub repo
12
Copy and use for RStudio
Things in Flow
Create new RStudio project
13
Things in Flow
RStudio is now connected with GitHub
14
Docker
Things in Flow
Docker: How it works
16
Traditional Virtual Machines:
Full OS for each VM
Docker:
OS code is partly shared
Things in Flow
The ideal Docker workflow
17
Source: docker.com
Things in Flow
Another take on the same
18
Source: terranillius.com
Things in Flow
Using Docker
• Install and start Docker for Mac (or Windows)
• Verify that Docker is running
• Search for Docker images on Docker Hub • … either via http://dockerhub.com • …or via Kitematic (installed as part of Docker for Mac)
• Start your first Docker container i.e. hello-world-nginx
• You are now running Docker locally
19
Things in Flow
Demo 2: Deploy Shiny App on local VM
• Start Docker for Mac (or Windows)
• Verify that Docker is now running
• Add Dockerfile to project
• Add docker-compose.yml file to project
• Open terminal via RStudio-link
• Run: docker-compose up -d
• Shiny App is now running on local Docker VM
• …and can be accessed via http://localhost
20
Things in Flow
Docker cheatsheetdocker run <container> <command>
docker run -v LOCAL_DIR:CONTAINER_DIR <container> <command>
docker exec -it <container_id> <command>
docker images
docker logs <container_id>
docker-compose up -d --build
docker-compose start
docker rmi [imageID]
docker ps [-a]
docker run ubuntu /bin/echo 'Hello world'
21
Things in Flow
Manage data in containers
• If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it’s best to create a named Data Volume Container, and then to mount the data from it.(from docker.com)
22
Things in Flow
Using Compose is basically a three-step process
1. Define your app’s environment with a Dockerfile so it can be reproduced anywhere.
2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.
3. Lastly, run docker-compose up and Compose will start and run your entire app.(from docker.com)
23
Things in Flow
Compose can mange the whole lifecycle
• Start, stop and rebuild services (start, stop, up, down)
• View the status of running services
• Stream the log output of running service (logs)
• Run a one-off command on a service (run)(from docker.com)
24
Things in Flow
Volume data preserved when containers are created
• Compose preserves all volumes used by your services. When docker-compose up runs, if it finds any containers from previous runs, it copies the volumes from the old container to the new container. This process ensures that any data you’ve created in volumes isn’t lost.(from docker.com)
25
Things in Flow
Moving Compose files between environments
• Compose supports variables in the Compose file. You can use these variables to customize your composition for different environments, or different users. See Variable substitution for more details.
• You can extend a Compose file using the extends field or by creating multiple Compose files. See extends for more details.(from docker.com)
26
From Local to Cloud
Things in Flow
Cloud Deploy: How it works
28
Source: circleci.com
Things in Flow
The Workflow
29
bit.ly/R-Docker-workflow
R-Docker Workflow for teams and small companies
Loca
lLo
cal V
M (D
ocke
r)C
loud
(Circ
leC
I/Dig
italO
cean
)
v1, Niels Ole Dam, 5/10-2016
Start
Client
GitHub, Docker Hub and Cloud Server (i.e. DigitalOcean, Amazon, Azure etc.)
http://thingsinflow.dk/
Show/Hide Messages
Develop R solution + test locally
Test passed?
NoTest/fix on local VM
Test passed?
NoSetup initial deploy to
Cloud
Deploy to local VM
Test/iterate with client
Test passed?
No
Initial solution
deployed
Build on local VM
docker-compose up
Build on Cloud server
Develop, test, verify and deploy
docker-compose up
1. 2. 3.
Setup auto deploy on CircleCI Solution
updated4.
Switch prod server
5.
6.
7.
GitHub Docker CircleCIRstudio Project
Git branch
bit.ly/R-Docker-workflow = clickabel version of this document
Docker Cheat Sheet:- docker run <container> <command>- docker run -v
LOCAL_DIR:CONTAINER_DIR <container> <command>
- docker exec -it <container_id> <command>- docker images- docker logs <container_id>- docker-compose up -d- docker-compose start- docker rmi [imageID]- docker ps [-a]- docker run ubuntu /bin/echo 'Hello world'
Request solution
Request change
Try out new frameworks on local VM
8.
Test php/
Spark?
Click on the blue numbers or doc icons for more info on that step/subject.
CircleCI
(or Jenkins or TravisCI etc.)
Things in Flow
CircleCI: How it works
• Connects to GitHub
• Listens for any changes on a specific GitHub repo
• Looks for a circle.yml script in the root of the GitHub repo
• Runs the circle.yml script every time the repo changes
• Can be used for automatical running of: • …code tests • …docker image builds for Docker Hub • …deployment(s) to Cloud server(s)
31
Things in Flow
Demo 3-4: Setup CircleCI for auto deploy
• CircleCI connects to GitHub and Cloud via environment variables (=secure because credentials are not shown in code)
• Environment variables could be: • $DOCKER_USER • $DOCKER_PASS • $DEPLOY_TOKEN • $DOCKER_EMAIL
• Test that the circle.yml script is working as planned • Now everything is set up for auto deploy on push to GitHub!
32
Things in Flow
Known issues with CircleCI
• Docker 1.12 is currently NOT supported by CircleCI
• Docker 1.10 is currently NOT installed out-of-the-box by CircleCI! (=compose must be installed explicitly)
• But should not be a too big issue
• …if it is, then use TravisCI (or own Jenkins etc.) instead
33
Things in Flow
Demo 5-6: Auto deploy to DigitalOcean server
• Make a code change to the RStudio project
• Commit and push to Master (GitHub)
• Wait 10 min – you’ll also get a mail
• A new version of the cloud server has now been build on DigitalOcean with a random IP
34
Things in Flow
Demo 7: Switch prod server and setup A record
• New cloud server has now been build
• Log into DigitalOcean controlpanel
• Switch the floating IP address from old to new server
• Requires: A subdomain (i.e. http://meetup.thingsinflow.dk) has been setup on your website (A Record)
35
Things in Flow
Docker Swarm Mode: How it works
• 30 sec demo:
• www.johnzaccone.io/3-node-cluster-in-30-seconds/
• Using docker-compose with docker swarm • https://docs.docker.com/compose/swarm/
• Swarm mode overview: • https://docs.docker.com/engine/swarm/
36
Bonus: Docker for local try outs
Things in Flow
Try out new frameworks safely
• Test out new frameworks in a local VM without: • …risking the health of your main computer • …messing with the initial configuration • …thinking too much about documenting your steps
• Quickly test out new integrations
38
Things in Flow
Demo 8: Try out some images
• hello-world-nginx
• Jenkins
• php
• Java
• MySQL
• Spark
• Hadoop
• …
39
Summing Things Up
Things in Flow
Summing up: Steps needed
• Connect RStudio to GitHub – see this link at Preferences -> Git/SVN
• Make your Shiny App
• Add docker-compose.yml and/or Dockerfile
• Test Docker setup locally
• Connect GitHub to Docker Hub
• Connect Docker Hub to CircleCI (or other CI service)
• Connect CircelCI to a cloud server (i.e. Amazon, DigitalOcean)
• Push to GitHub and watch the magic unfold
Tips: • Dockerfile and circle.yml: Use one-liners or shell script • Beware of absolute vs. relative paths in all scripts
41
Things in Flow
Stuff not touched today
• Partly update of individual containers on cloud server via GitHub and CircleCI (i.e. only on container at a time)
• Sharing of volumes between Docker container and local machine
• How to add a nice html frontend page (index.html) ;-)
42
Things in Flow
Key Take Aways
• Docker is production ready for teams and small companies
• Docker makes it possible to use the Continuous Integration (CI) paradigme for your R projects
• Docker with R is Awesome! :-)
43
Things in Flow
Links• Getting started with Docker:
• http://www.docker.com/products/overview • https://docs.docker.com/engine/getstarted/
• Manage Data in containers: • https://docs.docker.com/engine/tutorials/dockervolumes/#/creating-
and-mounting-a-data-volume-container • Docker Compose:
• https://docs.docker.com/compose/overview/ • Automated Docker Deployments with CircleCI:
• http://www.mikeheijmans.com/docker/2015/09/07/automated-docker-deployment-with-circle/
• Continuous Integration and Delivery with Docker-CircleCI: • https://circleci.com/docs/docker/
44
Things in Flow
Questions?
Things in Flow
Niels Ole Dam – [email protected] thingsinflow.dk – Mob. +45 2712 9314
Thanks! :-)CopenhagenR - useR Group, Meetup at ITU, Oct. 11 2016
This presentation can be found at thingsinflow.dk/R-Docker-workflow