Post on 25-Jun-2015
01
About me02
Andrey AdamovichBio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: andrey@aestasit.com
Linkedin: http://www.linkedin.com/in/andreyadamovich
Twitter: @aestasit
•••••
03
What's this presentation about?Our take on:
DevOps
Intrastructure Provisioning
Continuous Integration
Continuous Delivery
••••
04
TechnologiesGroovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
AWS - http://aws.amazon.com
•••••
05
Developers +Operations =
?06
Silos
07
Conflicts08
Risk
09
Agile
10
What is DevOps?
11
C.A.M.S.Culture : People over processes and tools. Software is made by and
for people.
Automation : Automation is essential for DevOps to gain quick
feedback.
Measurement : DevOps finds a specific path to measurement. Quality
and shared (or at least aligned) incentives are critical.
Sharing : Creates a culture where people share ideas, processes, and
tools.
•
•
•
•
12
It's not abouttools!
13
It's aboutculture!
14
We startedwith the tools!
15
Infrastructure as codeAutomate the provisioning and maintenance of servers:
Build from source control
Utilize existing tools
Ensure testability
•••
16
Configuration propagation
17
Configuration propagation
18
ChangesImagine manually uploading *.class files and repackaging JAR
directly on production servers when you have an urgent code change.
19
Deploymentis automatic!
20
And, so,should be...
21
infrastructureconfiguration
changes!22
No manualchanges!
23
Building an automation toolkitAutomation is key
We are JVM hackers
Fragmented ecosystem
•••
24
Initial toolsetGradle
Groovy
Ant
Python/WLST
Shell scripts
•••••
25
Required toolingInfrastructure connectivity
Infrastructure provisioning
Infrastructure virtualization
Infrastructure testing
••••
26
First Blood27
Ant + Gradleant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
28
Simple callant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
29
Next step: wrapper functiondef ssh(String command,
Properties props,
boolean failOnError = false,
String suCommandQuoteChar = "'",
String outputProperty = null) {
...
}
01.
02.
03.
04.
05.
06.
07.
30
Next step: wrapper functiondef scp(String file,
String remoteDir,
Properties props) {
...
}
01.
02.
03.
04.
05.
31
Task example Itask installFonts << {
forAllServers { props ->
ssh('yes | yum install *font*', props)
}
}
01.
02.
03.
04.
05.
32
Task example IItask uninstallNginx << {
forAllServers { props ->
ssh('/etc/init.d/nginx stop', props)
ssh('yes | yum remove nginx', props, true)
ssh('rm -rf /etc/yum.repos.d/nginx.repo', props)
ssh('rm -rf /var/log/nginx', props)
ssh('rm -rf /etc/nginx /var/nginx', props)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 33
DrawbacksNew connection each time
Excplicit repeating parameters
Complex scripts are hard to maintain
Tasks are not idempotent
••••
34
Sshoogr
35
Sshoogr featuresGroovy-based SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•••
36
Why Groovy?Groovy is perfect choice for scripting
Gradle build scripts are Groovy
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
•••••
37
Shameless plug
38
Sshoogr usage (import)@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.16')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
39
Sshoogr usage (defaults)defaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
verbose = true
showCommand = true
}
01.
02.
03.
04.
05.
06.
40
Sshoogr usage (connection)remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
41
Sshoogr usage (multi-line content)remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
42
Sshoogr usage (file copying)remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
43
Sshoogr usage (command result)def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 44
Sshoogr usage (shortcuts)if (ok('/usr/bin/mycmd')) {
...
}
if (fail('/usr/bin/othercmd')) {
...
}
01.
02.
03.
04.
05.
06.
45
Sshoogr usage (tunnels)tunnel('1.2.3.4', 8080) { int localPort ->
def url = "http://localhost:${localPort}/flushCache"
def result = new URL(url).text
if (result == 'OK') {
println "Cache is flushed!"
} else {
throw new RuntimeException(result)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 46
Sshoogr usage (prefix/suffix)prefix('sudo ') {
exec 'rm -rf /var/log/abc.log'
exec 'service abc restart'
}
suffix(' >> output.log') {
exec 'yum -y install nginx'
exec 'yum -y install mc'
exec 'yum -y install links'
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 47
Still problemsComplex scripts are still not easy to maintain
Scripts are usually not idempotent••
48
Puppet49
Why Puppet?More mature than competition
Large community
Readable DSL
Good acceptance from DEVs and OPs
No need to learn Ruby ;)
•••••
50
Puppet example
51
Puppet provisioning
52
Puppet provisioning
53
Puppet provisioning
54
Puppet provisioning
55
Puppet state management
56
Puppet state management
57
Puppet state management
58
Puppet modules
59
Puppet modules
60
Puppet modules
61
Sshoogr +Gradle +Puppet62
Upload modulestask uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
63
Upload modules ...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
64
Apply manifeststask puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 65
What we solved?Separated infrastructure state description and operations tasks
Scripts became more maintainable and idempotent••
66
In the meanwhile...We started developing complex/generic Puppet modules
Modules need proper testing
...on different platforms
•••
67
Do you test, right?How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
68
PUnit
69
PUnitSimple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
••••
70
PUnit example (derby)class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
71
PUnit example (derby)@Test
void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
72
PUnit example (derby)@Test
void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
73
Continuous integration
74
JenkinsDe-facto standard
Stable
There is a plugin for that!
•••
75
Jenkins build
76
Nextproblem?
77
ScalabilityHow do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
78
Amazon WebServices
79
Elastic Compute CloudMature
Great API
Virtual hardware variety
OS variety
••••
80
Gramazon
81
GramazonGroovy-based API for interacting with EC2
Integration with Gradle••
82
Gramazon example Itask startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09. 83
Gramazon example IItask terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
84
The flowStart instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
85
Next issue?86
Imgr
87
ImgrA tool for building images
Inspired by Packer••
88
SupportsShell
Puppet••
89
Configuration example
90
Summary91
Images, manifests, tasks
92
The big picture
93
Aetomation
94
ConclusionsReuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Infrastructure configuration is outsourcable now!
Automate!
••••••
95
Next steps?Create more documentation and examples
Add more DSL convience methods
Extend integration with Gradle
Add Windows connectivity/scripting support (groowin)
Define richer model for EC2 and potentially other clouds
Extend support for other provisioning tools
••••••
96
Readingmaterial
97
The Phoenix Project
98
Continuous Delivery
99
Release It
100
Programming Amazon EC2
101
Gradle in Action
102
Groovy 2 Cookbook
103
Technologies to followVagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
••••••
104
One morething...
105
It's all OpenSource!
106
Source codeSshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
107
Seekingcontributors!
108
Questions?109
Thank you!
110