EuroPython 2014 Devops Risk Mitigation

download EuroPython 2014 Devops Risk Mitigation

of 21

  • date post

    23-Aug-2014
  • Category

    Internet

  • view

    282
  • download

    4

Embed Size (px)

description

The (perceived) risk of the DevOps is that too many people get the right to "break" the platform. Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others. This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers. See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

Transcript of EuroPython 2014 Devops Risk Mitigation

  • DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro
  • Expensive Fix Cheap Fix Costs Of Finding Bugs In Production
  • Plan Design Budget Develop Test RUN DEV Plan Design Budget Develop Test RUNOPS TIME
  • Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked Buy Now button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
  • DevOps: Respect & Learning DEV learn from OPS to think about: Resources (CPU, RAM, Disk) Services (Start, Stop, Status) Dependencies (Start DB before App) Logfiles (Rotate, Remove) Disk Space Monitoring and Alarming ... OPS learn from DEV to think about: Incremental Improvement Infrastructure as Code Version Control System Coding (OO, Functions, Libraries ) Code Quality Unit & Integration Tests Test Automation ...
  • Untested = Broken
  • Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
  • Part of build process Syntax checks Scripts Config Files Data Files Unit tests for functions/libs Run program with test data Check result Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside HTTP calls Send emails Try to login Run tests from inside Remote Exec (rsh, ssh ) http://go.schapiro.org/rshpitfall Service Start, Stop & Status Modify server to create good & bad test scenarios Reboot Unit Tests System Tests
  • Unit Tests
  • %prep %setup -q %install install %{buildroot}/ install %{buildroot}/ %files %defattr(-,root,root,-) /...
  • BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
  • BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c " import yaml yaml.safe_load(open('config.yaml')) "
  • System Tests http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
  • Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job Check out source Run Unit Tests Create RPM Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job Check out source Run Unit Tests Create RPM Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
  • SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
  • HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server
  • VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
  • 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
  • Low Risk Lots of Fun http://go.schapiro.org/slides
  • Kontakt: Immobilien Scout GmbH Andreasstrae 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.