Post on 24-Jan-2015
description
01
About me02
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: andrey@aestasit.com
Linkedin: http://www.linkedin.com/in/andreyadamovich
Twitter: @aestasit
•••••
03
What's this presentation about?
Our take on:
DevOps
Intrastructure Provisioning
Continuous Integration
Continuous Delivery
••••
04
Technologies
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
AWS - http://aws.amazon.com
•••••
05
Developers +Operations =
?06
Silos
07
Conflicts08
Risk
09
Agile
10
What is DevOps?
11
C.A.M.S.
Culture : People over processes and tools. Software is made by and
for people.
Automation : Automation is essential for DevOps to gain quick
feedback.
Measurement : DevOps finds a specific path to measurement. Quality
and shared (or at least aligned) incentives are critical.
Sharing : Creates a culture where people share ideas, processes,
and tools.
•
•
•
•
12
It's not abouttools!
13
It's aboutculture and
process!14
But withouttools...
15
...it'sdefinitelyharder!
16
DevOps implyautomation!
17
DevOps implystructure!
18
Infrastructureas code
19
Infrastructure as code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize existing tools
Ensure testability
•••
20
Configuration propagation
21
Configuration propagation
22
Changes
Imagine uploading *.class files and repackaging JAR directly on
production servers when you have an urgent code change.
23
Deployment isautomatic!
24
And, so,should be...
25
infrastructureconfiguration
changes!26
No manualchanges!
27
Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
•••
28
Initial toolset
Gradle
Groovy
Ant
Python/WLST
Shell scripts
•••••
29
Required tooling
Infrastructure connectivity
Infrastructure provisioning
Infrastructure virtualization
Infrastructure testing
••••
30
First Blood31
Ant + Gradle
ant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
32
Simple call
ant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
33
Next step: wrapper function
def ssh(String command,
Properties props,
boolean failOnError = false,
String suCommandQuoteChar = "'",
String outputProperty = null) {
...
}
01.
02.
03.
04.
05.
06.
07.
34
Next step: wrapper function
def scp(String file,
String remoteDir,
Properties props) {
...
}
01.
02.
03.
04.
05.
35
Task example I
task installFonts << {
forAllServers { props ->
ssh('yes | yum install *font*', props)
}
}
01.
02.
03.
04.
05.
36
Task example II
task uninstallNginx << {
forAllServers { props ->
ssh('/etc/init.d/nginx stop', props)
ssh('yes | yum remove nginx', props, true)
ssh('rm -rf /etc/yum.repos.d/nginx.repo', props)
ssh('rm -rf /var/log/nginx', props)
ssh('rm -rf /etc/nginx /var/nginx', props)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.37
Drawbacks
New connection each time
Excplicit repeating parameters
Complex scripts are hard to maintain
Tasks are not idempotent
••••
38
Sshoogr
39
Sshoogr features
Groovy-based SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•••
40
Why Groovy?
Groovy is perfect choice for scripting
Gradle build scripts are Groovy
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
•••••
41
Shameless plug
42
Sshoogr usage (import)
@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.16')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
43
Sshoogr usage (defaults)
defaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
verbose = true
showCommand = true
}
01.
02.
03.
04.
05.
06.
44
Sshoogr usage (connection)
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
45
Sshoogr usage (multi-line content)
remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
46
Sshoogr usage (file copying)
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
47
Sshoogr usage (command result)
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.48
Sshoogr usage (shortcuts)
if (ok('/usr/bin/mycmd')) {
...
}
if (fail('/usr/bin/othercmd')) {
...
}
01.
02.
03.
04.
05.
06.
49
Sshoogr usage (tunnels)
tunnel('1.2.3.4', 8080) { int localPort ->
def url = "http://localhost:${localPort}/flushCache"
def result = new URL(url).text
if (result == 'OK') {
println "Cache is flushed!"
} else {
throw new RuntimeException(result)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.50
Sshoogr usage (prefix/suffix)
prefix('sudo ') {
exec 'rm -rf /var/log/abc.log'
exec 'service abc restart'
}
suffix(' >> output.log') {
exec 'yum -y install nginx'
exec 'yum -y install mc'
exec 'yum -y install links'
}
01.
02.
03.
04.
05.
06.
07.
08.
09.51
Still problems
Complex scripts are still not easy to maintain
Scripts are usually not idempotent
••
52
Puppet53
Why Puppet?
More mature than competition
Large community
Readable DSL
Good acceptance from DEVs and OPs
No need to learn Ruby ;)
•••••
54
Puppet example
55
Puppet provisioning
56
Puppet provisioning
57
Puppet provisioning
58
Puppet provisioning
59
Puppet state management
60
Puppet state management
61
Puppet state management
62
Puppet modules
63
Puppet modules
64
Puppet modules
65
Sshoogr +Gradle +Puppet
66
Upload modules
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
67
Upload modules
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
68
Apply manifests
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.69
What we solved?
Separated infrastructure state description and operations tasks
Scripts became more maintainable and idempotent
••
70
In the meanwhile...
We started developing complex/generic Puppet modules
Modules need proper testing
...on different platforms
•••
71
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
72
PUnit
73
PUnit
Simple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
••••
74
PUnit example (derby)
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
75
PUnit example (derby)
@Test
void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
76
PUnit example (derby)
@Test
void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
77
PUnit example (derby)
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.78
PUnit example (derby)
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
79
PUnit example (jenkins)
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
def driver = new HtmlUnitDriver(false)
driver.manage()
.timeouts()
.pageLoadTimeout(300, TimeUnit.SECONDS)
.implicitlyWait(30, TimeUnit.SECONDS)
driver.get("http://127.0.0.1:${localPort}/login")
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.80
PUnit example (jenkins)
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(By.name('j_password'))
input.sendKeys('123456')
input.submit()
...
01.
02.
03.
04.
05.
06.
07.
81
PUnit example (jenkins)
...
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.
presenceOfElementLocated (By.linkText('John Doe'))
...
}
}
01.
02.
03.
04.
05.
06.
07.
82
PUnit example (svn)
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1',
localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.
04.
05.
06.
07.
08.
09.83
PUnit example (svn)
...
// Setup credentials.
def authManager = SVNWCUtil.
createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.84
PUnit example (svn)
...
// Commit first revision.
ISVNEditor editor = repository.
getCommitEditor("Initial commit.", null)
editor.with {
openRoot(-1)
addFile('dummy.txt', null, -1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
01.
02.
03.
04.
05.
06.
07.
08.
09.85
PUnit example (svn)
...
def checksum = deltaGenerator.sendDelta('dummy.txt',
new ByteArrayInputStream("data".getBytes()),
editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09.86
PUnit example (svn)
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.
05.
87
Continuous integration
88
Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•••
89
Jenkins build
90
Nextproblem?
91
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
92
Amazon WebServices
93
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
••••
94
Gramazon
95
Gramazon
Groovy-based API for interacting with EC2
Integration with Gradle
••
96
Gramazon example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.97
Gramazon example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
98
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
99
Next issue?100
Imgr
101
Imgr
A tool for building images
Inspired by Packer
••
102
Supports
Shell
Puppet
••
103
Configuration example
104
Summary105
Images, manifests, tasks
106
The big picture
107
Aetomation
108
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•••••
109
Next steps?
Create more documentation and examples
Add more DSL convience methods
Extend integration with Gradle
Add Windows connectivity/scripting support
Define richer model for EC2 and potentially other clouds
Extend support for other provisioning tools
••••••
110
Readingmaterial
111
The Phoenix Project
112
Continuous Delivery
113
Release It
114
Programming Amazon EC2
115
Gradle in Action
116
Groovy 2 Cookbook
117
Technologies to follow
Vagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
••••••
118
One morething...
119
It's all OpenSource!
120
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
121
Seekingcontributors!
122
Questions?123
Thank you!
124