New Building a New Site with Ansible · 2019. 9. 17. · •Ansible Tower or Semaphore if you want...

Building a New Site with AnsibleEmanuele Simili, Gordon Stewart

and the Glasgow ScotGrid team

Background (or why bother?)

Glasgow Data Centre UpdateGareth Roy, HEPSysMan, 16/01/2018

Background (or why bother?)

Glasgow – Site ReportEmanuele Simili, HEPSysMan, 24/05/2019

Data Centre

• We obtained access at the start of August

• Migration in progress – will take until the spring

• Facility completion scheduled for mid-October

• We’re really excited…

• …but now we have to do all the things we said we’d do “once we have a data centre”

Background (or why bother?)• Rare opportunity to rebuild site from the ground up

• What works? What doesn’t?

• What do we like? What really annoys us?

• Simplifies migration to the data centre• Can commission and test new cluster, then move compute

resources gradually

• Minimises downtime

• New site is “remote”• Necessitates a change in approach

• It’s fun!

Brief introduction to Ansible

• First released 2012

• Ansible, Inc. acquired by Red Hat since 2015

• Agentless• No central controller• Connects via SSH or

WinRM

• Ansible Tower or Semaphore if you want a GUI

StackOverflow Developer Survey 2019

Why Ansible?• It’s simple and lightweight

• Configuration is self-documenting and readable• (In our opinion!)

• It appeals to our need to be control freaks

• It can be run from anywhere…• …although work is ongoing to make this a reality

• We can share expertise and outputs with PPE / PHAS

Find us online• A heavily-sanitised version of our Ansible roles and

playbooks can be found here:

https://github.com/uki-scotgrid-glasgow/ansible

• [ Generic health warning here ]

Provisioning• Ansible (configuration management) is actually step 2

• Step 1 (provisioning) involves an automated kickstart install over UEFI…• …or PXE, in the case of our older, more recalcitrant hardware

• Before setting Ansible loose, we assume a baseline system configuration:• Bare up-to-date CentOS 7 installation

• Network interfaces configured and up

• Routes set as necessary to provide Internet access

• SSH key permitting password-less access from Croquembouche

Terminology*• A task does something

• Install a package, write to a file

• A handler does something automatically• Restart a service later

• Tasks and handlers can be grouped in a role to do something useful• Install and configure a particular application

• Roles can be grouped into a playbook to do lots of things in an orchestrated fashion on a group of hosts• Set up an ARC CE

*Grossly oversimplified

Inventory

Add hoc commands• Can easily run commands on hosts or groups from

inventory

• A bit like pexec or pdsh

arc-ce.ymlargus.ymlceph-vanilla.ymlcondor-ce.ymlcondor-manager.ymldhcp.ymldisk-local.ymldns.ymldo_yumclean.ymlgitlab.ymlmonitoring.ymlnat.yml

perfsonar.ymlprovisioning.ymlservice.ymlsquid.ymlupdate-networking.ymlworknodes.yml

Playbooks• Typically one per class of system

• Some perform particular common tasks

- hosts: argus

roles:- common- ansible-prereq- time- yum-cron- ssh-authorized-keys- grid-security- hostcerts- firewall-zones- packages- argus- vo-atlasvo-dteamvo-gluexvo-opsvo-scotgrid

vars_files:- /etc/ansible/vars/infrastructure.yml- /etc/ansible/vars/secure.yml

Playbooks

ansible-prereqarc-cearguscommoncondorconntrack-basiccvmfsdhcpdnsmasqfirewall-zonesgitlabgrid-securityhostcertsipforwardipv6-disablenatnode_exporterpackagesperfsonarppepixieprometheusrepo-argusrepo-cvmfs

repo-egi-trustanchorsrepo-epelrepo-frontierrepo-globusrepo-htcondorrepo-nordugrid-6repo-umd4repo-wlcgrepo-xrootdsquidssh-authorized-keystimevo-atlasvo-dteamvo-gluexvo-opsvo-scotgridvo-templateworknodexrootd-serveryum-cron

Roles

[root@croquembouche roles]# ls –R1 xrootd-server/xrootd-server/:metataskstemplates

xrootd-server/meta:main.yml

xrootd-server/tasks:configure.ymlinstall.ymlmain.yml

xrootd-server/templates:auth_file.j2xrootd-standalone.cfg.j2

Roles

# ls –R1 xrootd-server/xrootd-server/:metataskstemplates




Rolesdependencies:

- { role: repo-xrootd }





Roles- include: install.yml

tags: xrootd

- include: configure.ymltags: xrootd





Roles- name: Install XRootD server

yum:name:

- xrootd- xrootd-client- xrootd-server

state: latest

- name: Identify internal and external interfacessite_facts_network:

- name: Assign external interface to appropriate zonefirewalld:permanent: yesimmediate: trueinterface: {{ ansible_facts['if_external'] }}zone: externalstate: enabled

when: ansible_facts['if_external'] != None

Extensibility• Ansible can be easily extended with custom modules

and plug-ins

• Use the language of your choice…• …although easiest with Python

#!/usr/bin/python

import subprocessfrom ansible.module_utils.basic import AnsibleModule

def run_module():# Seed the result dictresult = dict(changed = False,)

# Instantiate AnsibleModule objectmodule = AnsibleModule(argument_spec = {}, supports_check_mode = True)

# Obtain network configuration via 'ip' commandp = subprocess.Popen(['/usr/sbin/ip', '-oneline', 'addr', 'show'], stdout = subprocess.PIPE)(out, err) = p.communicate()

if_internal = Noneif_external = None

# Iterate over interfacesfor l in out.splitlines():

l = l.split()if l[2] == 'inet6':

continue

# Check IPv4 address (n.b. this is not portable, and assumes at most one internal and one external interface per host)name = l[1]ipv4 = l[3]if ipv4.startswith('10.1.'):

if_internal = nameelif ipv4.startswith('130.209.239.'):

if_external = name

result['ansible_facts'] = {}result['ansible_facts']['if_internal'] = if_internalresult['ansible_facts']['if_external'] = if_externalresult['message'] = 'Internal: {0} External: {1}'.format(if_internal, if_external)

# Return resultsmodule.exit_json(**result)

def main():run_module()

if __name__ == '__main__':main()

Extensibility

Some boilerplate to kick things off

Do whatever you need to do here...

(yes, this example is a bit of a fudge)

Construct a dictionary and dump the result out as JSON

Summary• New data centre is nearing completion

• Significant effort has been invested rebuilding site from ground up, both to aid migration and to improve operation

• Ansible roles available to show how we’ve approached things:• https://github.com/uki-scotgrid-glasgow/ansible

• Should be fully ensconced in DC by GridPP 44

• Did I mention that we have a data centre?

Final picture, I promise…

New Building a New Site with Ansible · 2019. 9. 17. · •Ansible Tower or Semaphore if you want...

Documents

Transcript of New Building a New Site with Ansible · 2019. 9. 17. · •Ansible Tower or Semaphore if you want...