Ansible Tower Administration Guide Release Ansible Tower 2.4.3 ...
With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git...
Transcript of With Ansible and Git Umut Bozkurt, 20.11...Standardizing Oracle Environment With Ansible and Git...
−Problem
− Increasing number of databases
− Increasing security requirements
− Consistent number of DBAs
−Solution
− Increasing number of DBAs
− Standardization
− Automation
30
40
50
60
70
80
90Oracle Database Size (TB)
About me…
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank4
−Umut Bozkurt
− Database Engineer
− Swiss National Bank
−More than 18 years of Oracle Database experience
− from database development to Oracle hardware
Oracle Infrastructure
−Current platform: IBM AIX, LPAR
−Next platform: Red Hat, VMware ESX
−Oracle technologies
− Single instance
− Multitenant
− Data Guard
− Automatic Storage Management
− Oracle Enterprise Manager
− Automation with shell scripts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank6
What is Ansible?
− Ansible is an automation language and engine
− Ansible is agentless and uses OpenSSH or WinRM
− Ansible terminology:
− Playbooks files contains plays, in YAML
− YAML is a data serialization standard that can be used in
conjunction with all programming languages
− Plays consists of tasks and ensures that managed hosts
are in a particular state
− Tasks calls modules
− Tasks runs sequentially
− Playbooks are executed from the control node
− Managed hosts are listed in the inventory
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank7
- name: httpd server installation
hosts: all
become: yes
become_user: root
tasks:
- name: httpd server is installed
yum:
name: httpd
state: present
- name: httpd is started and enabled
service:
name: httpd
state: started
enabled: true
What is Ansible Tower?
− Ansible Tower is a web console and REST API
− Ansible Tower is a commercial offering from Red Hat
− Ansible AWX is the upstream project of the Ansible Tower
− Ansible Tower features:
− Role based access control
− Job scheduling
− Workflows
− Credential management
− Logging and auditing
− Real time and historical job status reporting
− Notifications
− REST API
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank8
Ansible Tower Organization
−Credentials
− Git repository credentials
− Server credentials
− Projects
− Git connection
− Git credential
− Git branch
− Inventories
− Managed hosts
− Groups
− Variables
Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank9
Security Concept - Personal Users
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank10
Authentication
AD account 1 +
Smart Card certificate
Authentication
AD account 2 +
Smart Card certificate
Authentication
Unix account + RSA SecurID
Authorization
Database Server privileges
Change Management
># Database Server># ---------------> logged in as umut> sudo su - oracle
># Database Server># ---------------> logged in as umut> sudo su -
># Database Server># ---------------> logged in as umut> sudo su -
># Jump Host># ---------------
> ssh umut@dbsrv
Authentication
AD account 3
−Running Ad-Hoc commands only with the personal accounts
Security Concept - Jobs
− Scheduling jobs only with a technical user and a privileged
credential
− Scheduler user
− Only execute privileges on the assigned template
−Credential
− Encrypted and stored in the Ansible Tower database
− Can only be used from the Tower GUI or API
− Exclusive for the scheduler user
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank11
# Database Server# ---------------> cat authorized_users...from "10.24.23.11" ssh-rsa AKCB3NzaC1yc...
# Database Server# ---------------> cat sshd_config... AllowUsers [email protected]...
# Database Server# ---------------> sudo su -
Security Concept - Operations
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank12
− Execute privileges on a template allows running the template. Viewing or modifications are
not possible
− Surveys allows operations team or end users to run the templates with input parameters
Standardization
− IT standardization is a strategy for minimizing the IT
costs by keeping the hardware and software as
consistent as possible
−Our goal is to consistently push and (if changed)
overwrite both our scripts and configuration files to the
database servers
− This will lead us to
− easy automation
− increase security
− reduce human errors
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank14
># Development Server># ------------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
># Database Server># ---------------
Standardization with Ansible and Git
−Git
− Playbooks, inventory scripts
− Files to push to the servers
− LDAP
− Server and lifecycle information
−CMDB
− Oracle instance information
−Oracle Enterprise Manager
− Monitoring
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank15
># Database Server># ----------------
># Database Server># ----------------
># Database Server># ----------------
LDAP CMDB
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Git Projects
Files to push to the servers
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank16
oracle_distribution
oracle_automation
Scripts for automation, like playbooks,
dynamic inventory scripts,..
Protected branches cannot be deleted and cannot be force pushed
Inventory
−Managed hosts are defined in the inventory
−Hosts are organized by groups
− The inventory can be defined either
− in a static text file
− dynamically by scripts, from the external sources
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank17
# My Static Inventory# -------------------
[PROD]
dbserver01
dbserver02
dbserver03
[TEST]
dbserver05
dbserver06
[AVALOQ]
dbserver01
dbserver05
[SAP]
dbserver03
dbserver06
Instance Information in the Inventory
− In most cases, extending the inventory with the instance
information is helpful
− Instance information can be gathered either
− directly from the managed hosts (custom facts)
− from the external inventories
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank18
# My Static Inventory# -------------------
[PROD]
dbserver01 instance=AAAL HOME=/u01/...
dbserver02 instance=DB01 HOME=/u01/...
dbserver03 instance=PRQ HOME=/PRQ/...
[TEST]
dbserver05 instance=AAAQ HOME=/u01/...
dbserver06 instance=SRQ HOME=/SRQ/...
[AVALOQ]
dbserver01 instance=AAAL HOME=/u01/...
dbserver05 instance=AAAQ HOME=/u01/...
[SAP]
dbserver03 instance=PRQ HOME=/PRQ/...
dbserver06 instance=SRQ HOME=/SRQ/...
Ansible Facts
− Facts are
− variables that are automatically discovered on the
managed host
− collected before executing the first task
− Fact information can be accessed later in the tasks
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank19
PLAY [Install Oracle PDB] *************
TASK [Gathering Facts]***************
ok: [birs1z.snb.ch]
TASK [Create PDB] *******************
changed: [birs1z.snb.ch]
PLAY [Install Oracle PDB] *************
TASK [Gathering Facts]***************
ok: [server1.ubozkurt.com]
TASK [Create PDB] *******************
changed: [server1.ubozkurt.com]
Custom Facts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank20
−Custom facts
− allows us to gather additional information (like instance information)
from the managed hosts
− are either static files or scripts stored in the managed host
−Custom fact scripts must
− be executable
− have the extension ".fact”
− output in JSON
− be placed in /etc/ansible/facts.d directory on the managed host
>./local_script.fact
{"instances": [
{"name": "DBMS01CD","oracle_home": "…","version": "19c"
},{
"name": "DBMS02CD","oracle_home": "…","version": "18c"
}]
}
Dynamic Inventory
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank21
− Inventory scripts
− output in JSON
− mandatory arguments:
− --list
− --host <hostname>
− runs on the Ansible controller
− does not accept input parameters. But
you can work with environment
variables
> ./snb_invent.py --list
{“production": {
"hosts": ["prdserver1.snb.ch","prdserver2.snb.ch"
],"vars": {}
},“development": {
"hosts": ["devserver3.snb.ch"
]"vars": {}
}}
> ./snb_invent.py \--host prdserver2.snb.ch
{"instances": [
{"name": "DBMS01CD","oracle_home": "…“,"version": "19c"
},{
"name": "DBMS02CD","oracle_home": "…“,"version": "18c"
}]
}
Clone Git Repository to the Ansible Tower
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank22
># Ansible Tower>------------------
> ls -la tower_fs/oracle_distribution
.gitu01_app_oraclehome_oracle
- name: Clone Git project oracle_distribution
hosts: localhost
connection: local
gather_facts: false
tasks:
- name: Clone dev branch to Ansible Tower file system
git:
repo: 'http://ansitower:vk6z@gitlab/dbms/oracle_distribution.git'
dest: /tower_fs/
version: dev
force: yes
depth: 1
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank23
># Database Server>------------------
> ls -l /u01/app/oracle/snb
basenvbindbsetchistoryinstalllog -> /var/log/dbmsrcvsqltmp
- name: Oracle File Distribution
hosts: all
become: yes
become_user: oracle
gather_facts: true
gather_subset: min
tasks:
- name: Synchronize from the Tower file system to the managed host
synchronize:
src: "/tower_fs/oracle_distribution/u01_app_oracle/snb/{{ item }}"
dest: "/u01/app/oracle/snb/"
checksum: yes
delete: yes
times: yes
recursive: yes
rsync_opts: "--chown=oracle:dba"
loop:
- basenv
- bin
- install
- rcv
- sql
Distribute files from Ansible Tower to the Servers
Configuration files with Jinja2 Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank24
− A template contains variables which are replaced by their values when the template is
rendered
− Ansible uses Jinja2 templating
<title>{% block title %}{% endblock %}</title><ul>{% for user in users %}
<li><a href="{{ user.url }}">{{ user.username }}</a></li>{% endfor %}</ul>
listener.ora with Jinja2 Templates
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank25
# Jinja2 Template # -----------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={{ ansible_fqdn }})(PORT=1599)))
SID_LIST_LISTENER=(SID_LIST={% for instance in hostvars[inventory_hostname].instances %}(SID_DESC=(ORACLE_HOME={{ instance.oracle_home }})(SID_NAME={{ instance.name }})){% endfor %})
# Listener.ora# -----------------------------------------------------------------LISTENER=(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbserver.snb.ch)(PORT=1599)))
SID_LIST_LISTENER=(SID_LIST=(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS01CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home19c)(SID_NAME=DBMS02CD))(SID_DESC=(ORACLE_HOME=/u01/app/oracle/product/home18c)(SID_NAME=DBMS03CD)))
# Output of the Dynamic Inventory # Script# -------------------------------
> ./snb_invent.py --host dbserver.snb.ch{"instances": [{"name": "DBMS01CD","oracle_home": "…","version": "19c"
},{"name": "DBMS02CD","oracle_home": "…","version": "19c"
},{"name": "DBMS03CD","oracle_home": "…","version": "18c"
}]}
# Task in the playbook# ------------------------------------ name: creating listener.ora
template:src: "/tower_fs/oracle_distribution/db00_app_oracle/snb/templates/listener.j2"dest: "/u01/app/oracle/network/admin/listener.ora"owner: oraclegroup: dbamode: '0640'force: yes
Modules
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank27
- name: httpd server installation
hosts: all
become: yes
become_user: root
tasks:
- name: httpd server is installed
yum:
name: httpd
state: present
- name: httpd is started and enabled
service:
name: httpd
state: started
enabled: true
># Ansible Tower
># -------------
> cat /usr/lib/python2.7/site-packages/ansible/modules/system/service.py
#!/usr/bin/python
...
options:
name:
description:
- Name of the service.
type: str
required: true
state:
description:
- C(started)/C(stopped) are idempotent actions that will not run
commands unless necessary.
- C(restarted) will always bounce the service.
- C(reloaded) will always reload.
- B(At least one of state and enabled are required.)
type: str
choices: [ reloaded, restarted, started, stopped ]
...
Module Support
− Four types of modules:
− Core: Maintained and supported by Ansible
engineering team
− Network: Maintained and supported by Ansible
Network team
− Certified: Maintained by Red Hat partners. First
level contact is Red Hat
− Community: Not supported under an Ansible
engine subscription
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank28
Documentation of the synchronize module
−Organizations requires regulations about using the community modules
Oracle Modules
−One of the most popular community module for Oracle is
Ansible-Oracle-Modules, developed by Mikael Sandström
− Can be downloaded from GitHub
−Oracle also provides Oracle Cloud Infrastructure Ansible
modules
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank29
> ls ansible-oracle-modules
oracle_asmdgoracle_asmvoloracle_awroracle_datapatchoracle_dboracle_factsoracle_gi_factsoracle_grantsoracle_joboracle_jobclassoracle_jobscheduleoracle_jobwindoworacle_ldapuseroracle_opatchoracle_parameteroracle_pdboracle_privsoracle_profileoracle_redooracle_roleoracle_rsrc_consgrouporacle_servicesoracle_sqloracle_stats_prefsoracle_tablespaceoracle_user
− Automating database tasks without logging in to each
server
− Database software installation
− Database creation
− Database configuration
− Database patching and upgrade
− Database cloning
− ...
− Base for a self-service platform
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank30
># Database Server># ---------------
> Creating User
># Database Server># ---------------
> Cloning PDB
># Database Server># ---------------
> Creating Database
Self Service Portal
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Oracle Infrastructure Automation with Ansible
># Database Server># ---------------
> Creating User
># Ansible Tower ># Rest API># ---------------
curl -X GET …
Applying Release Update
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank31
- name: Start the instance and apply datapatch
shell: |
export ORACLE_HOME="{{ item.oracle_home }}"
export ORACLE_SID="{{ item.name }}"
$ORACLE_HOME/bin/sqlplus / as sysdba << EOF
startup
EOF
$ORACLE_HOME/OPatch/datapatch -verbose
register: shell_result
changed_when: >
( "'Installing patches...' in shell_result.stdout" ) or
( "'ORACLE instance started.' in shell_result.stdout" )
failed_when: "'ORA-' in shell_result.stdout"
with_items: "{{ hostvars[inventory_hostname]['instances'] }}"
when: item.oracle_home == "/db00/app/product/db19c/db00"
- name: Copy RU from the control host to the servers and unzip
unarchive:
copy: yes
src: "/ansible_fs/p30125133_190000_Linux-x86-64.zip"
dest: "/db00/app/snb/tmp_for_apply_psu"
owner: oracle
group: dba
- name: Apply RU with Ansible-Oracle-Modules
oracle_opatch:
oracle_home: "/db00/app/oracle/product/db19c/db00"
patch_base: "/db00/app/snb/tmp_for_apply_psu/30125133"
opatch_minversion: '12.2.0.1.17'
conflict_check: yes
stop_processes: no
state: present
Ansible Execution
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank32
># Ansible Tower># ---------------
> ansible-playbook db.yml -e "oracle_sid=orcl"
># Managed Host># ----------------
> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory
SSH
−Most of the modules requires python installed on the managed host. Modules can also
have additional requirements
− All playbooks are executed as awx user on the control node
># Managed Host># ----------------
> 1. create directory in ~/.ansible/tmp/…> 2. copy module in the created directory> 3. execute> 4. send results in JSON format> 5. remove temporary directory
Developing Ansible Modules
− You can use any language; however python is the most suitable
− Ansible modules strive for idempotency:
− You can run a playbook on the same host multiple times. When your systems are in the correct
state, the playbook should not make any changes
− Idempotency is not always possible especially when you are calling external programs, like
oracle setup or grid infrastructure root scripts
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank33
> mkdir /u01/app
> If (! -d /u01/app); then mkdir /u01/app ; fi
x
Simple Create/Remove Directory Module with bash and python
−Default custom module directory in Ansible Tower is
/usr/share/ansible/plugins/modules
− Ansible calls custom scripts with a parameter file
−Output should be a JSON object
− “failed” should be true in case of a failure; exit value is
not enough
− “changed” key is for idempotency
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank34
> ./my_bash_script.ksh /tmp/variables
{ "changed" : false,"msg": "ERROR:cannot create directory…","failed": true }
- name:create directory with a bash modulemy_bash_script:
dirname: "/tmp/new_dir"state: present
#!/usr/bin/ksh
source $1
echo ${dirname}echo ${state}
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank35
#!/usr/bin/bash
source $1
typeset -l state=${state:-present}
if !([ "${state}" = "absent" ] || [ "${state}" = "present" ]) then
echo "{ \"failed\": true,"
echo "\"msg\": \"ERROR: state must be absent or present\"}"
exit 1
fi
if [ "${dirname}" = "" ] ; then
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: please provide dirname \" }"
exit 1
fi
#!/usr/bin/python
import os
def main():
module = AnsibleModule(
argument_spec = dict(
dirname = dict(required=True, type='str'),
state=dict(default='present',choices=['present','absent'])
)
)
dirname = module.params["dirname"]
state = module.params["state"]
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank36
if [ "${state}" = "absent" ] && [ -d "${dirname}" ] ; then
vRetMsg=$( rm -rf ${dirname} 2>&1 )
if [ $? -eq 0 ] ; then
echo "{ \"changed\": true }"
else
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: ${vRetMsg}\" }"
exit 1
fi
elif [ "${state}" = "present" ] && [ ! -d "${dirname}" ] then
vRetMsg=$( mkdir ${dirname} 2>&1 )
if [ $? -eq 0 ] ; then
echo "{ \"changed\": true }"
else
echo "{ \"failed\": true, "
echo " \"msg\": \"ERROR: ${vRetMsg}\" }"
exit 1
fi
else
echo " { \"changed\": false }"
fi
exit 0
if state == 'absend' and os.path.exists(dirname):
try:
os.rmdir(dirname)
except OSError as err:
module.fail_json(msg="OS error: {0}".format(err))
else:
module.exit_json(changed=True)
elif state == 'present' and not os.path.exists(dirname):
try:
os.mkdir(dirname)
except OSError as err:
module.fail_json(msg="OS error: {0}".format(err))
else:
module.exit_json(changed=True)
else:
module.exit_json(changed=False)
from ansible.module_utils.basic import *
if __name__ == '__main__':
main()
Simple Create/Remove Directory Module with bash and python
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank37
TASK [Create directory] ************************************************
changed: [srv1.ubozkurt.com]
TASK [Create directory second time] ************************************************
ok: [srv1.ubozkurt.com]
TASK [Create directory without having permission on the fs] ************************************************
fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: mkdir: cannot create directory ‘/umut_new_dir’: Permission denied"}
...ignoring
TASK [Invalid input parameter] ************************************************
fatal: [srv1.ubozkurt.com]: FAILED! => {"changed": false, "msg": "ERROR: state must be absent or present"}
...ignoring
tasks:
- name: Create directory
my_bash_script:
dirname: "/tmp/umut_new_dir"
state: present
- name: Create directory second time
my_bash_script :
dirname: "/tmp/umut_new_dir"
state: present
- name: Create directory without having permission on the fs
my_bash_script:
dirname: "/umut_new_dir"
state: present
ignore_errors: true
- name: Invalid input parameter
my_bash_script:
dirname: "/tmp/umut_new_dir"
state: "invalid_value"
ignore_errors: true
Summary
− Ansible is a simple, powerful automation engine
− There are many use cases for automating daily tasks
−Role based access control helps to integrate Ansible Tower in to the organizations
−Organization wide usage requires a good security concept
17/11/2019 Standardizing Oracle Environment | Umut Bozkurt | © Swiss National Bank38