Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow...
Transcript of Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow...
![Page 1: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/1.jpg)
Large-Scale Software DevelopmentTools and Methods
![Page 2: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/2.jpg)
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
2
![Page 3: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/3.jpg)
Scenarios - configuration• Configuration files • Setup scripts • Container environments • Virtual Machines
3
![Page 4: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/4.jpg)
Version control — Git
4
![Page 5: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/5.jpg)
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
5
![Page 6: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/6.jpg)
Version control - GIT
6
![Page 7: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/7.jpg)
Sample Disorganized Project• “Hey, Anders, could you send me a copy of those changes you
made last Tuesday?” • “Ola, this function doesn’t work anymore. Did you change
something?” • “Sorry, I can’t seem to find those old classes. I guess you’ll just
have to re-implement them.” • “OK, we’ve all been working hard for the last week. Now let’s
integrate everyone’s work together.”
7
![Page 8: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/8.jpg)
What is version control?Basic functionality:
keep track of changes made to files (allows roll-backs) merge the contributions of multiple developers
Benefits: facilitates backups increased productivity (vs manual version control) encourages experimentation helps to identify/fix conflicts makes source readily available – less duplicated effort
8
![Page 9: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/9.jpg)
Additional benefitsAccountability
who wrote the code? do we have the rights to it?
Support software engineering hooks for peer reviews
Software branches different versions of software need to be maintained, ensure bug fixes shared
Record Keeping Commit logs may tie to issue tracking system or be used to enforce guidelines
9
![Page 10: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/10.jpg)
More BenefitsSupport Distribution of Work
Telecommuting, outsourcing, open-source projects Use in conjunction with “good communication habits” – via email etc.
Rapid Development (XP/Agile) Supports frequent refactoring Helps automate frequent system builds
10
![Page 11: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/11.jpg)
11
![Page 12: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/12.jpg)
The Big Picture
http://blog.mikepearce.net/2010/05/18/the-difference-between-git-pull-git-fetch-and-git-clone-and-git-rebase/
![Page 13: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/13.jpg)
Branching
• Gitseescommitthisway…• Branchannotateswhichcommitweareworkingon
13
![Page 14: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/14.jpg)
Branching
14
![Page 15: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/15.jpg)
15
![Page 16: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/16.jpg)
16
![Page 17: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/17.jpg)
17
![Page 18: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/18.jpg)
18
![Page 19: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/19.jpg)
19
![Page 20: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/20.jpg)
20
![Page 21: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/21.jpg)
21
![Page 22: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/22.jpg)
22
![Page 23: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/23.jpg)
Merging
• Whatdowedowiththismess?– Mergethem
23
![Page 24: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/24.jpg)
Merging
• Stepstomergetwobranch– Checkoutthebranchyouwanttomergeonto– Mergethebranchyouwanttomerge
24
![Page 25: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/25.jpg)
25
![Page 26: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/26.jpg)
26
![Page 27: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/27.jpg)
27
![Page 28: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/28.jpg)
28
![Page 29: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/29.jpg)
29
![Page 30: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/30.jpg)
30
![Page 31: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/31.jpg)
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
31
![Page 32: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/32.jpg)
Package management systems
32
![Page 33: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/33.jpg)
Dependency management
33
![Page 34: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/34.jpg)
Dependency management issues• Is a request to modify the current software component graph
satisfiable? – Are additions compatible with other components? – Are deletions safe with respect to other dependencies?
• Given a component, determine versions of other components we can safely rely on
34
![Page 35: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/35.jpg)
Dependency management as satisfiability
35
(a | b | c) & (d | e | f) ... = TRUE
(a | b | c) & (-c) & (-b | -a) ... = TRUE
A requires B provided by B1, B2, B3Rule: (-A | B1 | B2 | B3)
A conflicts with B provided by B1, B2, B33 Rules: (-A | -B1), (-A | -B2), (-A | -B3)
Rules
![Page 36: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/36.jpg)
Dependency management issues• Y depends on X >= 1.8. X makes binary incompatible changes from
v. 1.9 to v. 2.0... • Can components be installed from local sources as well as from
remote? • Should OS-specific dependency management or language-specific
be used?
36
![Page 37: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/37.jpg)
Software package management systems
37
Name Environment Format
NuGet .Net CLR XML
Gradle JVM XML
dpkg/APT Linux Ar archive
Rubygems Ruby Ruby
MSI Windows In-file DB
BSD Ports OS X/Linux/BSD Makefile
...
![Page 38: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/38.jpg)
Maven
38
mvn -h
Lifecycles
Clean
Default
Site
validatecompiletestpackageverifyinstalldeploy
![Page 39: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/39.jpg)
Maven example 39
Olas-MacBook-Pro:java-petclinic olale$ mvn graphwalker:test[INFO] Scanning for projects...[ … ][INFO] >>> graphwalker-maven-plugin:3.4.2:test (default-cli) > [graphwalker]test-compile @ java-petclinic >>>[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:generate-sources (generate-sources) @ java-petclinic ---[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:validate-models (default-cli) @ java-petclinic ---[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:generate-sources (default-cli) @ java-petclinic ---[INFO][INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ java-petclinic ---[INFO] Using 'UTF-8' encoding to copy filtered resources.[INFO] Copying 10 resources[INFO][INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ java-petclinic ---[INFO] Nothing to compile - all classes are up to date[INFO][INFO] --- graphwalker-maven-plugin:3.4.2:validate-test-models (default-cli) @ java-petclinic ---
![Page 40: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/40.jpg)
Maven — structure
40
mvn archetype:generate -DgroupId=com.mycompany.app -DartifactId=my-app -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
![Page 41: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/41.jpg)
Maven — Configuration
41
<parent> <groupId>org.graphwalker.example</groupId> <artifactId>graphwalker-example</artifactId> <version>3.4.2</version> </parent>
<artifactId>java-petclinic</artifactId>
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.1</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> <plugin> <groupId>org.graphwalker</groupId> <artifactId>graphwalker-maven-plugin</artifactId> <version>${project.version}</version> <!-- Bind goals to the default lifecycle --> <executions> <execution> <id>generate-sources</id> <phase>generate-sources</phase> <goals> <goal>generate-sources</goal> </goals> </execution> </executions> </plugin> </plugins> </build>
![Page 42: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/42.jpg)
Automation and languages for … development42
ASM
C
Java
Compiler
JVM
OS
Buildtools
Testautomation,packaging,…CIworkautomationMultipledevelopers/languages/components
![Page 43: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/43.jpg)
Jenkins 43
Workflowautomationtool
![Page 44: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/44.jpg)
Jenkins 44
Workflowautomationtool-pipelines
node{//<1>stage('Build'){//<2>sh'make'//<3>}
stage('Test'){sh'makecheck'junit'reports/**/*.xml'//<4>}
stage('Deploy'){sh'makepublish'}}
Groovy(JVM-basedlanguage)
![Page 45: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/45.jpg)
Travis CI 45
![Page 46: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/46.jpg)
Automation and languages for … deployment
46
Singlefile,structurednon-programminglanguage.Requirementsinstalled,updated,startedmanually.Deploymentthroughmanualscripts
Programminglanguage Automationofsetup
Chef/Puppet/Ansible/Salt/Terraform
Isolationofcomponents
Xen/SolarisZones,Docker/Chroot
![Page 47: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/47.jpg)
Chef/Puppet/Ansible/Salt/Terraform/…• Automates setup of environments for development/deployment • Introduces programming languages for your infrastructures
• Declarative (Terraform/Puppet) - What is the desired end-state? • Imperative (Chef/Ansible) - How do you achieve it?
47
![Page 48: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/48.jpg)
48
![Page 49: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/49.jpg)
Jenkins
49
![Page 50: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/50.jpg)
Scenarios - actions1. Single developer, multiple changes — Version control system 2. Many developers, multiple changes — Distributed version control system 3. Many groups of developers, multiple changes — Package management
system 4. Building artefacts based on multiple files with dependencies — build scripts 5. Conducting multiple actions with inter-dependencies on multiple files … —
Flexible build system 6. Automatically sensing changes and conducting such actions based on
changes — Continuous integration tools
50
![Page 51: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/51.jpg)
Containers (Docker in particular)
51
![Page 52: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/52.jpg)
Why containers
52
![Page 53: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/53.jpg)
Market View: Evolution of IT1995 2015
Runningonanyavailablesetof
physicalresources(public/private/virtualized)
Assembledbydevelopersusingbestavailable
services
Thinapponmobile,tabletThick,client-serverapp
onthickclient
Well-definedstack:-O/S-Runtime-Middleware
MonolithicPhysical
Infrastructure
![Page 54: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/54.jpg)
Challenges2015
Howtoensureservicesinteractconsistently,avoid
dependencyhell
Howtomigrate&scalequickly,ensurecompatibility
HowtoavoidnXndifferentconfigs
Runningonanyavailablesetof
physicalresources(public/private/virtualized)
Assembledbydevelopersusingbestavailable
services
Thinapponmobile,tablet
![Page 55: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/55.jpg)
Static website
Web frontend
User DB
Queue Analytics DB
Background workers
API endpoint
nginx 1.5 + modsecurity + openssl + bootstrap 2
postgresql + pgv8 + v8
hadoop + hive + thrift + OpenJDK
Ruby + Rails + sass + Unicorn
Redis + redis-sentinel
Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs
Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client
Development VM
QA server
Public Cloud
Disaster recovery
Contributor’s laptopProduction Servers
The ChallengeMultip
licity
ofS
tacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly?
![Page 56: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/56.jpg)
Results in N X N compatibility nightmareStatic website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster
Public Cloud
Contributor’s laptop
Customer Servers
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
![Page 57: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/57.jpg)
Perhaps A useful analogy…
![Page 58: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/58.jpg)
Multip
licity
ofG
oods
Multip
ilicityof
metho
dsfo
rtran
sportin
g/storing
DoIworryabouthow
goodsinteract(e.g.coffeebeansnextto
spices)
CanItransportquicklyandsm
oothly(e.g.from
boattotraintotruck)
Cargo Transport Pre-1960
![Page 59: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/59.jpg)
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
? ? ? ? ? ? ?
Also an NxN Matrix
![Page 60: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/60.jpg)
Multip
licity
ofG
oods
Multip
licity
of
metho
dsfo
rtransportin
g/storing
DoIworryabouthow
goodsinteract(e.g.coffeebeansnextto
spices)
CanItransportquicklyandsm
oothly(e.g.from
boattotraintotruck)
Solution: Intermodal Shipping Container
…in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another
A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.
![Page 61: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/61.jpg)
This eliminated the NXN problem…
![Page 62: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/62.jpg)
and spawned an Intermodal Shipping Container Ecosystem
• 90%ofallcargonowshippedinastandardcontainer• Orderofmagnitudereductionincostandtimetoloadandunloadships• Massivereductioninlossesduetotheftordamage• Hugereductioninfreightcostaspercentoffinalgoods(from>25%to<3%)à massiveglobalizations• 5000shipsdeliver200Mcontainersperyear
![Page 63: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/63.jpg)
Static website Web frontend User DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’s laptop
Docker is a shipping container system for code Multip
licity
ofStacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly
…that can be manipulated using standard operations and run consistently on virtually any hardware platform
An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…
![Page 64: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/64.jpg)
Static website Web frontend User DB Queue Analytics DB
Development VM
QA server Public Cloud Contributor’s laptop
Or…put more simplyMultip
licity
ofStacks
Multip
licity
of
hardware
environm
ents
Production Cluster
Customer Data Center
Doservicesandappsinteract
appropriately?
CanImigrate
smoothlyandquickly
Operator: Configure Once, Run Anything
Developer: Build Once, Run Anywhere (Finally)
![Page 65: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/65.jpg)
Static website
Web frontend
Background workers
User DB
Analytics DB
Queue
Development VM QA Server Single Prod
ServerOnsite Cluster
Public Cloud
Contributor’s laptop
Customer Servers
Docker solves the NXN problem
![Page 66: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/66.jpg)
66
What is Docker?Docker is an open-source project that automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating system–level virtualization on Linux.
[Source: en.wikipedia.org]
![Page 67: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/67.jpg)
67
Docker: Namedocker [naut.]: der Dockarbeiter, der Hafenarbeiter
Source: leo.org
• Provide a uniformed wrapper around a software package: «Build, Ship and Run Any App, Anywhere» [www.docker.com]
– Similar to shipping containers: The container is always the same, regardless of the contents and thus fits on all trucks, cranes, ships, ...
[www.docker.com]
![Page 68: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/68.jpg)
68
Docker vs. Virtual Machine
Source: https://www.docker.com/whatisdocker/
![Page 69: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/69.jpg)
69
Docker Technology
• libvirt: Platform Virtualization • LXC (LinuX Containers): Multiple isolated
Linux systems (containers) on a single host • Layered File System
[Source: https://docs.docker.com/terms/layer/]
![Page 70: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/70.jpg)
70
Docker History
• 2013-03: Releases as Open Source • 2013-09: Red Hat collaboration (Fedora, RHEL, OpenShift) • 2014-03: 34th most starred GitHub project • 2014-05: JAX Innovation Award (most innovative open
technology)
![Page 71: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/71.jpg)
71
Run Platforms• Various Linux distributions (Ubuntu, Fedora, RHEL, Centos,
openSUSE, ...) • Cloud (Amazon EC2, Google Compute Engine, Rackspace) • 2014-10: Microsoft announces plans to integrate Docker
with next release of Windows Server
![Page 72: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/72.jpg)
72
Hello WorldSimple Command - Ad-Hoc Container • docker run ubuntu echo Hello World – docker images [-a]
– docker ps –a
![Page 73: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/73.jpg)
73
Terminology - Image• Persisted snapshot that can be run
– images: List all local images – run: Create a container from an image and
execute a command in it – tag: Tag an image – pull: Download image from repository – rmi: Delete a local image
• This will also remove intermediate images if no longer used
![Page 74: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/74.jpg)
74
Terminology - Container• Runnable instance of an image
– ps: List all running containers – ps –a: List all containers (incl. stopped) – top: Display processes of a container – start: Start a stopped container – stop: Stop a running container – pause: Pause all processes within a container – rm: Delete a container – commit: Create an image from a container
![Page 75: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/75.jpg)
Container cid4
Container cid3
75
Image vs. ContainerBase Image ubuntu:latest
Container cid1
run
Container cid1
cmd à new state
New Image iid1
commit
base image
Container cid2
run
![Page 76: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/76.jpg)
76
Dockerfile• Create images automatically using a build script: «Dockerfile» • Can be versioned in a version control system like Git or SVN,
along with all dependencies • Docker Hub can automatically build images based on
dockerfiles on Github
![Page 77: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/77.jpg)
77
Dockerfile Example• Dockerfile:
– FROM ubuntu ENV DOCK_MESSAGE Hello My World ADD dir /files CMD ["bash", "someScript"]
• docker build [DockerFileDir]
• docker inspect [imageId]
![Page 78: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/78.jpg)
78
Mount Volumes• docker run –ti –v /hostLog:/log ubuntu
• Run second container: Volume can be shared – docker run –ti --volumes-from firstContainerName ubuntu
![Page 79: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/79.jpg)
79
Publish Port• docker run –t –p 8080:80 ubuntu nc –l 80
– Map container port 80 to host port 8080 – Check on host: nc localhost 8080
• Link with other docker container – docker run -ti --link containerName:alias ubuntu
– See link info with set
![Page 80: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/80.jpg)
80
Around Docker• Docker Images: Docker Hub • Vagrant: «Docker for VMs» • Automated Setup
– Puppet, Chef, Ansible, ...
![Page 81: Large-Scale Software Development - IDATDDE06/lectures/lecture2.pdf · Jenkins 44 Workflow automation ... Chef/Puppet/Ansible/Salt/Terraform Isolation of components Xen/Solaris Zones](https://reader031.fdocuments.us/reader031/viewer/2022020204/5b5beb297f8b9a905c8ef6d7/html5/thumbnails/81.jpg)
81
Docker Hub• Public repository of Docker images
– https://hub.docker.com/ – docker search [term]
• Automated: Has been automatically built from Dockerfile – Source for build is available on GitHub