backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to...
Transcript of backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to...
![Page 1: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/1.jpg)
Protecting your OpenStack cloud with an automated backup and recovery strategy
Carlos Camacho GonzalezSenior Software EngineerRed Hat
November 14, 2018
Dan MacphersonPrincipal Technical WriterRed Hat
![Page 2: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/2.jpg)
INSERT DESIGNATOR, IF NEEDED2
● Introduction● Defining the strategy● Backup and Restoring the Undercloud● Backup and Restoring the Overcloud● Challenges and Ideas
Agenda
![Page 3: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/3.jpg)
INSERT DESIGNATOR, IF NEEDED3
● How did we meet?● Fast forward upgrades● Problems we’re trying to solve
Introduction
![Page 4: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/4.jpg)
INSERT DESIGNATOR, IF NEEDED4
Protect against maintenance tasks failures (Undercloud, Overcloud control plane)Protect user space (Trilio, Freezer)
Backup categories
User workload
Backend services
Configurationfiles
Logfiles
Databases
![Page 5: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/5.jpg)
INSERT DESIGNATOR, IF NEEDED5
Goal: Ensure you can restore the Undercloud and the Overcloud controllers no matter
what… and all automatically!
![Page 6: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/6.jpg)
INSERT DESIGNATOR, IF NEEDED
Defining Backup Strategies for Individual Services
6
![Page 7: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/7.jpg)
INSERT DESIGNATOR, IF NEEDED7
Database (Non-HA)For example: backing up and restoring the undercloud.
Backup:● Run the mysqldump command
Restore:● Create a new database● Start mariadb● Increase the packet size● Restore data from .sql files
![Page 8: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/8.jpg)
INSERT DESIGNATOR, IF NEEDED8
Database (HA)Backup:
● Select an idle node● Backup the database● Backup the grants
Restore:● Disable VIP access to the database (iptables)● Stop Galera● Temporarily disable replication● Create a new database on each node● Set database permissions (root, clustercheck)● Synchronize the nodes● Enable replication● Start Galera● Import database and grants● Restore VIP access (iptables)
![Page 9: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/9.jpg)
INSERT DESIGNATOR, IF NEEDED9
MongoDBUsed for Telemetry storage in Newton
Backup:● mongodump● https://docs.mongodb.com/manual/reference/progr
am/mongodump/
Restore:● mongorestore● https://docs.mongodb.com/manual/reference/progr
am/mongorestore/
![Page 10: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/10.jpg)
INSERT DESIGNATOR, IF NEEDED10
RedisUsed as an object store for services. TripleO overclouds use it for Telemetry object storage.
“Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete.”
- https://redis.io/topics/persistence
Backup● Save the current state (redis-cli bgsave)● Copy the /var/lib/redis/dump.rdb
Restore● Stop Redis● Copy dump.rdb back to /var/lib/redis/● Start Redis
![Page 11: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/11.jpg)
INSERT DESIGNATOR, IF NEEDED11
Pacemaker ConfigurationRestore previous pacemaker configuration.
Backup:● Config backup command (pcs config backup pacemaker_backup)● Creates an archive file with configuration
Restore:● Stop the cluster (pcs cluster stop --all)● Restore config (pcs config restore pacemaker_controller_backup.tar.bz2)● Start the cluster (pcs cluster start --all)
![Page 12: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/12.jpg)
INSERT DESIGNATOR, IF NEEDED12
SwiftSwift object data as files. Usually part of a filesystem backup.
Backup:● Backup object files on each node (usually in /srv/node)● Don’t forget the xattrs (Swift object metadata)● Backup ringfiles and configuration (/etc/swift)
Restore:● Restore each node’s object files (usually to /srv/node)● Don’t forget the xattrs (Swift object metadata)● Restore ringfiles and configuration (/etc/swift)● Restart swift
Always important to include any xattrs option for rsync or tar commands:
# tar --xattrs ...# rsync --xattrs ...
![Page 13: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/13.jpg)
INSERT DESIGNATOR, IF NEEDED13
Backup relevant directories in your filesystem. You might need to restore a particular piece of configuration at some point.
Recommended directories:● /etc/● /var/lib/<service>/ (e.g. glance, cinder, heat, etc)● kolla config (e.g. /var/lib/config-data)● /srv/node/ (don’t forget xattrs!)● /var/log/● /root/ (contains .my.cnf for root access to database)● Your cloud admin user directory. In TripleO:
○ /home/stack for the undercloud○ /home/heat-admin for the overcloud
Filesystem Backup
![Page 14: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/14.jpg)
INSERT DESIGNATOR, IF NEEDED
Undercloud Backup and Restore
14
![Page 15: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/15.jpg)
INSERT DESIGNATOR, IF NEEDED15
Backing up the Undercloud
Create snapshots.Virtual node?
Backup the resources required to restore it back to a consistent state.Baremetal node?
OpenStack < Queens
Manual backups based on either bash or
Ansible.
OpenStack >= Queens
TripleO CLI option “openstack undercloud
backup”
![Page 16: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/16.jpg)
INSERT DESIGNATOR, IF NEEDED16
Backing up the Undercloud
Manual steps:
openstack undercloud backup [--add-path ADD_FILES_TO_BACKUP] [--exclude-path EXCLUDE_FILES_TO_BACKUP]
openstack undercloud backup --add-path /etc/ \ --add-path /var/log/ \ --add-path /root/ \ --add-path /var/lib/glance/ \ --add-path /var/lib/docker/ \ --add-path /var/lib/certmonger/ \ --add-path /var/lib/registry/ \ --add-path /srv/node/ \ --exclude-path /home/stack/
![Page 17: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/17.jpg)
INSERT DESIGNATOR, IF NEEDED17
Backing up the Undercloud
CLI driven:
mysqldump --opt --single-transaction --all-databases > /root/undercloud-all-databases.sql
sudo tar --xattrs --ignore-failed-read -cf \ UC-backup-`date +%F`.tar \ /root/undercloud-all-databases.sql \ /etc \ /var/log \ /root \ /var/lib/glance \ /var/lib/docker \ /var/lib/certmonger \ /var/lib/registry \ /srv/node \ /home/stack
![Page 18: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/18.jpg)
INSERT DESIGNATOR, IF NEEDED18
Restoring the Undercloud
Strategy?● Restore the snapshot or nuke the node and install from scratch [1]
Reasons?● Transaction history might be hard to rollback after an upgrade● Single node no HA, easy to reinstall
How to do it?● Restore the configuration files● Restore the certificates files● Restore the databases● Run: `openstack undercloud install`
[1]:https://docs.openstack.org/tripleo-docs/latest/install/controlplane_backup_restore/03_undercloud_restore.html
![Page 19: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/19.jpg)
INSERT DESIGNATOR, IF NEEDED
Overcloud Backup and Restore
19
![Page 20: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/20.jpg)
INSERT DESIGNATOR, IF NEEDED20
Overcloud Backup and Restore Strategy
● Composable and agnostic automated backup and restore system● Ansible role - ansible-role-openstack-operations [1]● Foundational ansible tasks [2]
○ Allows you to set an external backup server and automatically configure it○ Bootstrap node assignment○ Ansible synchronize module (rsync wrapper)○ Provides temporary SSH access to nodes○ Tasks for database backup○ Tasks for database restore (containerized HA)○ Tasks to validate the database
● Future goals:○ More services (Pacemaker, Redis, Swift, etc)○ Different backend architectures (Non-HA, non-containerized)
[1] http://git.openstack.org/cgit/openstack/ansible-role-openstack-operations/[2] https://review.openstack.org/#/c/604439/
![Page 21: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/21.jpg)
INSERT DESIGNATOR, IF NEEDED
---- name: Initialize backup host hosts: "{{ backup_hosts | default('backup') }}[0]" Tasks:
- import_role: name: ansible-role-openstack-operations
tasks_from: initialize_backup_host
- name: Backup MySQL database hosts: "{{ target_hosts | default('mysql') }}[0]" vars:
backup_server_hostgroup: "{{ backup_hosts | default('backup') }}" tasks:
- import_role: name: ansible-role-openstack-operations
tasks_from: validate_galera- import_role:
name: ansible-role-openstack-operations tasks_from: enable_ssh- import_role:
name: ansible-role-openstack-operations tasks_from: backup_mysql
- import_role: name: ansible-role-openstack-operations
tasks_from: disable_ssh
21
Backing up the overcloud
![Page 22: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/22.jpg)
INSERT DESIGNATOR, IF NEEDED
---- name: Initialize backup host hosts: "{{ backup_hosts | default('backup') }}[0]" tasks:
- import_role: name: ansible-role-openstack-operations
tasks_from: initialize_backup_host
- name: Restore MySQL database on galera cluster hosts: "{{ target_hosts | default('mysql') }}" vars:
backup_server_hostgroup: "{{ backup_hosts | default('backup') }}" tasks:
- import_role: name: ansible-role-openstack-operations tasks_from: set_bootstrap
- import_role: name: ansible-role-openstack-operations tasks_from: enable_ssh
- import_role: name: ansible-role-openstack-operations tasks_from: restore_galera
- import_role: name: ansible-role-openstack-operations
tasks_from: disable_ssh- import_role:
name: ansible-role-openstack-operations tasks_from: validate_galera
22
Restoring the overcloud
![Page 23: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/23.jpg)
INSERT DESIGNATOR, IF NEEDED23
Overcloud restore demo
![Page 24: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/24.jpg)
INSERT DESIGNATOR, IF NEEDED24
Trilio
User workloads
Freezer
![Page 25: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/25.jpg)
INSERT DESIGNATOR, IF NEEDED25
● Testing.● Adapting the tasks to several versions and services.● Maintenance over new releases.
Challenges
![Page 26: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/26.jpg)
INSERT DESIGNATOR, IF NEEDED26
● Including the ansible tasks per service configuration template.● Create an additional repository to store the backup/restore
workflow.● Composable backups.● Each squad testing their own backup/restore methodology.● Create a new CLI option to backup the Overcloud controllers?
○ openstack overcloud backup --controllers● TripleO UI options?
Ideas
![Page 27: backup and recovery strategy Protecting your OpenStack cloud …€¦ · Copy dump.rdb back to /var/lib/redis/ Start Redis. 11 INSERT DESIGNATOR, IF NEEDED Pacemaker Configuration](https://reader033.fdocuments.us/reader033/viewer/2022043019/5f3be504e24f3418421632d9/html5/thumbnails/27.jpg)
THANK YOUplus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews