How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL...
Transcript of How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL...
![Page 1: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/1.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
How can I backup my PostgreSQL databases?
![Page 2: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/2.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
custom scripts, pg_backrest, WAL-G,
pg_probackup, Barman, ..?
![Page 3: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/3.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
As technical people we have a tendency to focus
on the means (tools)
![Page 4: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/4.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Means without goalsare useless
![Page 5: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/5.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
What are the business continuity goals of my
PostgreSQL infrastructure?
![Page 6: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/6.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
With clear goalsany tool is fine
![Page 7: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/7.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Gabriele Bartolini
Barman in action
PGConf.eu 2019 - October 16
![Page 8: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/8.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
About myself● Open Source passionate and programmer since 1995● First time with Postgres in 1997, regular from ~2000● Co-Founder of ITPUG and PostgreSQL Europe● With 2ndQuadrant since 2008
○ Head of Global Support● Co-Founder and developer of Barman● Lean and DevOps practitioner
![Page 9: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/9.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
GoalsRecovery Point Objective (RPO)“How much data the business can afford to lose”
Recovery Time Objective (RTO)“How long it takes to restore the business service”
“From 0 to 100: Business continuity with PostgreSQL”:https://www.youtube.com/watch?v=-EuVjj3zqE8
![Page 10: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/10.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Incremental approach● Keep our goals in mind (RPO and RTO)● Focus on Disaster Recovery (DR)● Incrementally build a solid DR solution for PostgreSQL
based on Barman● What better approach than a practical one?
![Page 11: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/11.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Barman and PostgreSQL 12● Native● Transparent
○ postgresql.auto.conf for PostgreSQL 12■ With recovery.signal and standby.signal
○ recovery.conf for PostgreSQL 8.3 -> 11● Available from 1 Aug 2019
○ When PostgreSQL 12 was still in beta○ Request originated from a support case
![Page 12: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/12.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Playground used in this talkhttps://github.com/2ndquadrant-it/ansible-postgresql-barman-playground
4 machine Vagrant playground environment with Ansible Playbooks for provisioning of PostgreSQL and Barman
Available under GNU GPL 3
After the talk explore the playbook and “play” with it
![Page 13: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/13.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
![Page 14: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/14.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
The “Beatles” cluster● 4 CentOS pre-configured virtual machines:
○ Paul: PostgreSQL 12 + barman-cli on 192.168.33.10○ John: PostgreSQL 12 + barman-cli on 192.168.33.11○ George: Barman 2.10dev on 192.168.33.12○ Ringo: Barman 2.10dev on 192.168.33.13
![Page 15: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/15.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
What Ansible did for us● Basic PostgreSQL configuration
○ PGDATA initialised with data checksums○ Archive mode enabled
■ archive_command = /bin/true○ Syslog destination for logs○ Enable md5 access from 192.168.33.x in pg_hba○ Empty database for pgbench
● Paul and John have a running PostgreSQL instance● SSH key exchange between postgres and barman users
![Page 16: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/16.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Asciinema● I have recorded casts using Asciinema
○ https://asciinema.org● I will now be playing some casts, not all of them
○ See link at the end of the presentation
![Page 17: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/17.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
PgbenchAll commands have been executed with pgbench simulating workload in background.
![Page 18: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/18.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
What we will be doing today
paul
george
john
ringo
sync (RPO = 0)
![Page 19: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/19.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
![Page 20: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/20.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Hands-on now
![Page 21: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/21.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
There’s more …
paul
george
john
ringo
![Page 22: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/22.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
There’s more● Backup frequency (cron for “barman backup”)● Retention policies
○ Redundancy (number of backups)○ Point of Recoverability (based on time)
● Archive command (barman-wal-archive)● Streaming backup● Point In Time Recovery (define a target)● Monitoring, monitoring, monitoring● Backup from standby
![Page 23: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/23.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Future● Direct cloud storage (starting with AWS S3) - 2.10
○ barman-cloud-wal-archive○ barman-cloud-wal-restore○ barman-cloud-backup○ barman-cloud-recover
● Integration of the above with Barman - 2.11/3.0?● Multi-tier setups, based on retention policies:
○ Tier 2: Local compressed backups○ Tier 3: Cloud storage
![Page 24: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/24.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
DR Through the BeatlesI’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t do that! I should have known better, I have no backups! It’s been a hard day’s night ...Help! Please get back my Postgres database! I’ve got a feeling: I’ve lost all my data. I’ll cry instead.Recovery is now a long and winding road. Do you want to know a secret? I have learnt my lesson now. We can work it out, it is going to be a revolution. I will now take backups eight days a week.Oh, Here comes the sun! And with geo-redundancy and cloud storage it is going to be here, there and everywhere. From me to you: now I feel fine.
Sincerely yours,The fool on the hill (You: “All you need is love Barman”)
![Page 25: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/25.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
FeedbackPlease leave your feedback about my presentation, thanks!
https://2019.pgconf.eu/f
![Page 26: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/26.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Asciinema recordings● https://asciinema.org/a/udbWdTB4MqAthUhHlL5FP5lJw● https://asciinema.org/a/UGX6VPOm6BBeYvb7M6wYH03on● https://asciinema.org/a/hPiH7ZEaJux83aS6STaF56qX5● https://asciinema.org/a/wK8MMh1zvjbylQKvAbQEkvIQZ● https://asciinema.org/a/Mes4u4FktkVtjFbeln7NybIzY● https://asciinema.org/a/g9VZvMg5gqZUt9l8cn1ziJJX2● https://asciinema.org/a/XK0kRXmE9I4oFULXQtdmoOlD4● https://asciinema.org/a/YPGrH0GGZg7fNxmbtMo6W1IZk● https://asciinema.org/a/WUiCvtnDH8PFEcR5fSsrBtBHi
![Page 27: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/27.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Thanks to● Barman development team
○ They are all here today● For Ansible automation:
○ Rubens Souza○ Anna Bellandi
● Benjamin Zander for Rule #6○ https://www.youtube.com/watch?v=M-HG6X6fpBM
![Page 28: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/28.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Pass by our booth!
![Page 29: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/29.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
Thank you!Questions?
pgbarman.org - docs.pgbarman.org
Gabriele BartoliniTwitter: @_GBartolini_
![Page 30: How can I backup my PGConf.eu 2019 PostgreSQL databases? · I’m down. Yesterday my PostgreSQL cluster had a storage problem. I can’t bring it up. Don’t let me down! You can’t](https://reader034.fdocuments.us/reader034/viewer/2022050100/5f3fce8eecdfa66ff257a480/html5/thumbnails/30.jpg)
https://www.2ndQuadrant.com
PGConf.eu 2019Milan, October 16th - Gabriele Bartolini
@_GBartolini_ #pgconfeu
LicenseAttribution 4.0 International (CC BY 4.0)
You are free to:
● Share — copy and redistribute the material in any medium or format● Adapt — remix, transform, and build upon the material for any
purpose, even commercially.
The licensor cannot revoke these freedoms as long as you follow the license terms.