Automatically deploy Schema Changes with Ansible and Liquibase · ansible galaxy : community roles...
Transcript of Automatically deploy Schema Changes with Ansible and Liquibase · ansible galaxy : community roles...
Automatically deploy Schema Changes with Ansible and Liquibase
Robert Marz
8th June 2018 Robert Marz © its-people @RobbieDatabee
Senior Technical Architectwith database centric view of the worldClient
Portfolio Manager Database Technologies
Blog Editorits-people
Active Member Database Communityin charge of Cloud topicsDOAG
@RobbieDatabee [email protected]
Robert Marz
8th June 2018 Robert Marz © its-people @RobbieDatabee
Agile and DevOps – Database Challenges
8th June 2018 Robert Marz © its-people @RobbieDatabee
Agile meets Databases
Agi
le
Short Development cycles
Many Iterations
Frequent Schema Changes
8th June 2018 Robert Marz © its-people @RobbieDatabee
DevOps Requirements for Databases
De
vOp
s
ContinousIntegration
Rolling Releases Back and Forward
Mutliple (DB-) Servers
8th June 2018 Robert Marz © its-people @RobbieDatabee
Database Challenges
Dat
abas
eC
hal
len
ges DB
Schemas
DevOps Bottleneck
Scripts create
modify
revert
Existing Data
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase – OverviewDeploy, Rollback
Track, Manage, Document
DDL, Sources, Data
DB Changes
Oracle
MySQL, PostgreSQL, SQL Server, DB2 & more
Supported Databases
Change sets / Change logs
File formats
YAML, JSON, XML, SQL
Changelog Tables inside DBs
Incremental Changes
Commercial Product: Datical
Java based Open Source
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible in a Nutshell
Age
ntl
ess
IT A
uto
mat
ion Open Source Based on python
by RedHat
Parallel execution
Linux / OS Xvia ssh
Windows PowerShell Remoting
Ansible Playbooks
describe the desired state
YAML – Human and Machine readable
8th June 2018 Robert Marz © its-people @RobbieDatabee
YAML & JSON: two related file formats
databaseChangeLog:- changeSet:
id: 2author: nvoxlandchanges:- addColumn:
tableName: personcolumns:- column:
name: usernametype: varchar(8)
{ "databaseChangeLog": [{ "changeSet": {
"id": "2","author": "nvoxland","changes": [{ "addColumn": {
"tableName": "person","columns": [{ "column": {
"name": "username","type": "varchar(8)"
}}]
}}]
}}]
}
8th June 2018 Robert Marz © its-people @RobbieDatabee
JSON
{ "databaseChangeLog": [{ "changeSet": {
"id": "2","author": "nvoxland","changes": [{ "addColumn": {
"tableName": "person","columns": [{ "column": {
"name": "username","type": "varchar(8)"
}}]
}}]
}}]
}
JSO
N:
Java
Scr
ipt
Ob
ject
No
tati
on Data: “Key“: “Value“
Brackets: {} - Groupings
[] - Arrays
Schemaless
8th June 2018 Robert Marz © its-people @RobbieDatabee
YAML
databaseChangeLog:- changeSet:
id: 2author: nvoxlandchanges:- addColumn:
tableName: personcolumns:
- column:name: usernametype: varchar(8)
YAM
L:Y
AM
L A
in’t
Mar
kup
Lan
guag
e Humanreadable
structureddata format
Data Types Scalar (Key: Value)
Sequences (Arrays)
Mappings
GroupingWhitespace isimportant
Brackets as in JSON
JSON can be valid YAML
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible – The Basics
Only on Control Machine
Python 2 or 3
Windows not supported
Packages are available
yum install ansible
apt-get install ansible
…
Installation
ansible
ansible-playbook
ansible-console
REPL for debugging
ansible-vault
Encrypts sensitive Files
…
Command Line Interface
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible InventoryIn
ven
tory
lists
the
Ho
sts
An
sib
leca
nC
han
geDefault location /etc/ansible/hosts
File formats ini
YAML
Grouping
Static or dynamic
Variables per Host
Per Group
connection types ssh (default)
local, docker
mail.example.comjumper ansible_port=5555 ansible_host=192.0.2.50
[webservers]foo.example.com http_port=80 maxRequestsPerChild=808bar.example.com http_port=303 maxRequestsPerChild=909www[01:50].example.com
[dbservers]one.example.comtwo.example.comthree.example.comdb-[a:f].example.com
[dbservers:vars]ntp_server=ntp.atlanta.example.comproxy=proxy.atlanta.example.comhalon_system_timeout=30self_destruct_countdown=60escape_pods=2
[CoolApp:children]dbserverswebservers
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible Playbooks (1/3)---- hosts: stredax.devremote_user: roottasks:- name: Determine old APEX dirsshell: ls -1d /app/apex/{5*,4*,o*} 2>/dev/nullignore_errors: Trueregister: apex_dirs
- name: Remove old APEX dirsfile:path: "{{ item }}"state: absentwith_items: "{{ apex_dirs.stdout_lines }}"
- name: unzip apex and ordsunarchive:src: ./apex_ords.tar.gzdest: /app/apex
- name: Link ordsfile:path: /app/apex/ordssrc: /app/apex/ords.{{ ords_version }}state: link
- name: Change Oracle PWDsscript: change_ords_pwds.shremote_user: orastred
Pla
ybo
ok
Task Sequence
single steps
describe desired state
YAML (default)
Modules Abstract System tasks
Over 1300 predefined
Write your own
Variable values
Defined in Playbook
Facts gathered from Hosts
Passed from Inventory
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible Playbooks (2/3)P
layb
oo
kVariables Jinja2 templating
Filters available
Loops Iterate: {{ item }}
Define: with_items
Task Results (register)
---- hosts: stredax.eremote_user: roottasks:- name: Determine old APEX dirsshell: ls -1d /app/apex/{5*,4*,o*} 2>/dev/nullignore_errors: Trueregister: apex_dirs
- name: Remove old APEX dirsfile:path: "{{ item }}"state: absent
with_items: "{{ apex_dirs.stdout_lines }}"- name: unzip apex and ordsunarchive:src: ./apex_ords.tar.gzdest: /app/apex
- name: Link ordsfile:path: /app/apex/ordssrc: /app/apex/ords.3.0.10state: link
- name: Change Oracle PWDsscript: change_ords_pwds.shremote_user: orastred
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible Playbooks (3/3)---- hosts: webserversvars:- docroot: /var/www/stredax/publictasks:- name: Install Nginxapt: pkg: nginxstate: installedupdate_cache: true
register: nginxinstallednotify:- Start Nginx
- name: Create Web Rootwhen: nginxinstalled|successfile: dest: {{ docroot }} mode: 775state: directoryowner: www-datagroup: www-data
notify:- Reload NginxP
layb
oo
k
ConditionsWhen result| success/failed/skipped
"foo" == "bar"
Notify
HandlersTasks
Executed when notified
handlers:- name: Start Nginxservice: name=nginx state=started
- name: Reload Nginxservice: name=nginx state=reloaded
8th June 2018 Robert Marz © its-people @RobbieDatabee
Ansible RolesRoles organize multiple related
tasks and data
predefined folder structure:roles/rolename/…
ansible executes file main.ymlautomatically, if present
ansible galaxy :community roles repository
Vars with default Values
defaults
Files to be copied unchanged
files
Playbook Handlers
handlers
Dependencies
meta
Playbook Tasks
tasks
Content with Variables, e.g. config files. Will be proccessed by Jinja2 template enginge
templates
Variables
vars
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase – a closer LookLi
qu
ibas
eInstallation Download
Liquibase
unpack
Requirements: JDK 1.7 or higher
JDBC driver
Command Line Interface
liquibase update[SQL]
rollback[SQL]
Diff
dbDoc
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase CLI ParametersP
aram
ete
rsRequired --changeLogFile=<path
--username=<value>
--password=<value>
--url=<value>
--driver=<jdbc.driver.ClassName>
liquibase.properties
working dir
Parameter Default Values
8th June 2018 Robert Marz © its-people @RobbieDatabee
Defining Changes in Liquibase: ChangelogC
han
gelo
gscontain attributes
preConditions
changesets
nestable include
keepingTrack
Table databasechangelog
databaseChangeLog:- preConditions:dbms type: oraclerunningAs:
username: ROBBIE
- include:file: robbie-changeset-1.0.yamlrelativeToChangelogFile: true
- include:file: robbie-changeset-1.1.yamlrelativeToChangelogFile: true
- include:file: robbie-changeset-2.0.yamlrelativeToChangelogFile: true
8th June 2018 Robert Marz © its-people @RobbieDatabee
Defining Changes in Liquibase: ChangesetC
han
gese
tscontain Attributes
Actual changes
Rollback
identified ID
Author
Changelog Filename
runOnChange Excutes on change
Checksum in Table
PL/SQL code
databaseChangeLog:- changeSet:id: TabStockTicker-1author: robbiechanges:
- createTable:tableName: StockTickercolumns:- column:name: idtype: numberautoIncrement: trueconstraints:primaryKey: truenullable: false
- column:name: symboltype: varchar2(50)
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase ChangesdatabaseChangeLog:- changeSet:id: TabTab1-1author: robbiechanges:- createTable:tableName: Tab1columns:- column:name: idtype: number
rollback:- dropTable:
tableName: Tab1- changeSet:id: TabTab1-2author: robbiechanges:- dropTable:tableName: Tab1
rollback:changeSetId: TabTab1-1changeSetAuthor: robbie
Ch
ange
sBasic Operations
Bundled
With rollback
Example createTable
SQL / SQLFile
Extensions Oracle
Build your own
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase Workflow
Add changesetto changelog
file
Runliquibaseupdate
Commit changelog file
to sourcecontrol
Create new changelog file
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase Procedure for the developer
Editcreate new local changeSet
TestTest the changeSet SQL: Run liquibase update
ChangeChange your Application: Edit the AppCode
TestTest application code & database change together
CommitCommit changeSet and application code.
8th June 2018 Robert Marz © its-people @RobbieDatabee
Liquibase Best PracticesUse a master changelog
One change per changeset
PL/SQL Code with runOnChange=”true“
IDs are literals – use them
Document changesets with comments
Always think about rollback
Leverage Liquibase to manage your Reference Data
8th June 2018 Robert Marz © its-people @RobbieDatabee
Using Liquibase with Ansible
see example on the right
Deploy Liquibase to Targets
File Module: Transport changelogs
Shell Module: run liquibase
Execute liquibase update on target
Local_action with jdbc URLS
See Blog Post
Run Liquibase from Control Host
let ansible create ssh tunnels
Run Liquibase as local action
connecting to localhost
See Blog Post
ssh Tunnels
---- name: Create Liquibase directoryfile: name: /opt/liquibasestate: directorymode: 0755
- name: Install Liquibaseunarchive:src:
https://github.com/liquibase/liquibase/releases/download/liquibase-parent-{{ liquibase_version}}/liquibase-{{ liquibase_version }}-bin.tar.gz
dest: /opt/liquibasecopy: nomode: 0755creates: /opt/liquibase/liquibase
8th June 2018 Robert Marz © its-people @RobbieDatabee
Demo
8th June 2018 Robert Marz © its-people @RobbieDatabee
AlternativesPo
pu
lar
Alt
ern
ativ
es Liquibase Flyway
Ansible
Chef
Puppet
Terraform
8th June 2018 Robert Marz © its-people @RobbieDatabee
Want to know more?
• Homepage
• Official Documentation
• Tutorial: Change Happens by Blaine Carter
Liquibase
• Homepage
• Official Documentation
• Free Webinar
Ansible
• Tutorial
• Tutorial Learn X in Y Minutes
• Reference Card
• YAML 1.2 Spec
YAML
8th June 2018 Robert Marz © its-people @RobbieDatabee
Conclusion
Ansible issimple but
powerfull IT-Automation
Liquibase isSource
Control forYour
Database
OvercomeDatabaseDevOps
Bottleneck
PLEASE
DO
TRY THIS
AT HOME
its-people GmbH
Frankfurt Tel. 069 2475 2100
Hamburg Tel. 040 2360 8808
Köln Tel. 0221 1602 5204
München Tel. 089 5484 2401
its-people ERP Beratungsgesellschaft mbH
Frankfurt Tel. 069 2475 1980
www.its-people.de [email protected]
we make the difference
www.its-people.de
Questions ?
Herzlichen Dank für Ihre Aufmerksamkeit