Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment...
Transcript of Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment...
![Page 1: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/1.jpg)
Edward Colone, Eric L. Boyd, University of Michigan
![Page 2: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/2.jpg)
●○
■
● U-M Network Overview
● perfSONAR Deployment Challenges
● perfSONAR Deployment Automation
● POC Case Study: perfSONAR SC19 deployment
○ SC19 network
○ Timeline
● Next Steps
2
![Page 3: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/3.jpg)
●○
■3
![Page 4: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/4.jpg)
●○
■
● 8 core perfSONAR Nodes
● 2 satellite campus testpoints
● Multiple 1GE and 10GE mobile diagnostic testpoints
● Esmond Data Archive
● MadDash Dashboard
● pSconfig schedule publisher
● perfSONAR Web Admin
4
![Page 5: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/5.jpg)
●○
■
● Major manual upgrade of a 12 node cluster: perfSONAR 3.5 / CentOS 6 → perfSONAR 4.0 / CentOS 7○ Five UNIX sysadmins○ Two days to complete
● Version control done manually - environment skewed● Hard to coordinate system patching and software upgrade
responsibilities between groups● Staff time cost for node addition scaled linearly with each
node added
5
![Page 6: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/6.jpg)
●○
■
● Testpoints and Toolkits must be registered with Data Archivers
● Dashboards read Data Archivers
● Testpoints, Toolkits, and Dashboards read published schedules
● PWA User account provisioning
6
![Page 7: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/7.jpg)
●○
■
Archiver
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
7
![Page 8: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/8.jpg)
●○
■
● perfSONAR authors and supports an official Ansible master Playbook and Roles for component provisioning
● End users bootstraps machines with base OSes, user accounts
● Ansible Idempotency allows for continuously scalable infrastructure
● Ansible is agentless, uses SSH - no extra security overhead
● Open source industry standard solution
8
![Page 9: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/9.jpg)
●○
■
● https://github.com/perfsonar/ansible-playbook-perfsonar
● Simple instructions and examples to manage your entire environment
● Quick and easy bootstrapping for all Roles
● Script to create multiple Inventories
● Abstract individual Roles away from the end user
● No previous Ansible experience required
9
![Page 10: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/10.jpg)
●○
■
● Group Variables allow global definitions for fast mass configuration.
● Per-Host variables take precedence for specific host configuration and tuning
● Host file allows for easy inventory management and resource visualization.
● Inventories can be stored in source control for change management and disaster recovery.
● Multiple Inventories can be easily used for multiple Dev and Prod instances.
●●●
10
![Page 11: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/11.jpg)
●○
■
● https://github.com/perfsonar/ansible-role-perfsonar-archive
● https://github.com/perfsonar/ansible-role-perfsonar-installer
● https://github.com/perfsonar/ansible-role-perfsonar-maddash
● https://github.com/perfsonar/ansible-role-perfsonar-testpoint
● https://github.com/perfsonar/ansible-role-perfsonar-toolkit
● https://github.com/perfsonar/ansible-role-perfsonar-psconfig-publisher
● https://github.com/perfsonar/ansible-role-perfsonar-psconfig-web-admin
psconfig - Web Admin
Installer
Testpoint
Toolkit
Archive MadDash psconfig - publisher
11
![Page 12: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/12.jpg)
●○
■12
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
Archiver
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Download Playbook / Bootstrap Roles / Configure Inventory
![Page 13: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/13.jpg)
●○
■
Archiver
pS Maddash
Test Points
Publisher
PWA Admin
Download Playbook / Bootstrap Roles / Configure Inventory
pS Maddash
Test Points
Sched. Publisher
PWA Admin
Manual Deployment
Intermediate UNIX admin / Novice perfSONAR administrator / Full Configuration / 2-6 Test Points
13
Archiver
Automated Deployment
![Page 14: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/14.jpg)
●○
■14
![Page 15: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/15.jpg)
●○
■
perfSONAR TestPoints
● Virtual infrastructure included:○ Ansible Controller○ Esmond Data Archiver○ MadDash Dashboard○ JSON Schedule Publisher
Administrative VMs
● Physical Testpoints managed entirely by Ansible● Vendor managed testpoints deployed in booths
seamlessly integrated in to overall system.● Automated Mesh deployment reduced sources of
error and introduced time savings.
15
![Page 16: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/16.jpg)
●○
■
● Standard, project-supported playbooks
● Additional specialized playbooks developed for site-specific enhancements
● Scripts to automate deployment on Ansible controller
● Pre-configured virtual infrastructure before the conference and physical infrastructure on-demand.
● Provisioned all production systems at SC
● Near one-touch deployment
● Used Vagrant as proof-of-concept to test configuration in a virtual environment before the conference
16
![Page 17: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/17.jpg)
●○
■
● Systems required little manual setup
● Minimal OS
● Ansible user account (scripted to minimize errors)
● New systems added to inventory and provisioned as they were brought online
● Allowed testing of partial deployment instead of having to wait for everything
● Adjustments to systems applied consistently
● Significant labor savings left time to work on things needing attention
●
17
Best SCinet perfSONAR deployment in recent years
![Page 18: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/18.jpg)
●○
■
● Ansible support for more optional tools and tests
(GridFTP, SNMP, nmap, etc.)
● Enhanced Vagrant support for virtual development
and proof of concept environments
● More customizations for MadDash, PWA, etc.
● Enhanced automated environment troubleshooting
18
![Page 19: Edward Colone, Eric L. Boyd, University of Michigan · Version control done manually - environment skewed ... perfSONAR authors and supports an official Ansible master Playbook and](https://reader033.fdocuments.us/reader033/viewer/2022042804/5f5b5b6ca70060069d11bfae/html5/thumbnails/19.jpg)
●○
■
● Antoine Delvaux
● Mark Feit
● Ivan Garnizov
● perfSONAR Development Team
● U-M Networking Staff
● Katarina Thomas
19