Languages
Pages
Legal
DevOps Risk Mitigationwww.immobilienscout24.de
Berlin | 23.07.2014 | Schlomo SchapiroSystems Architect, Open Source Evangelist
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
Test Driven Infrastructure
@schlomoschapiro
ExpensiveFix
CheapFix
Costs Of Finding Bugs In Production
PlanDesignBudget
Develop Test
RU
NDEV
PlanDesignBudget
Develop Test RUN OPSTIME
PlanDesignBudget
Develop Test
PlanDesignBudget
Develop Test RUN
Proxyconfig
Databaseborked
„Buy Now“button broken
MTA dropall mail
Load BalancerConfiguration
Everything costsonly 0 €
Login possiblewithout password
NFS notavailable
DB Replicationstopped
No adsshown
BrokenCSS / JS
Tomcat won'tstart/stop
Service usernot defined
sudoersinvalid
DevOps: Respect & Learning
DEV learn from OPS to think about:
● Resources (CPU, RAM, Disk)● Services (Start, Stop, Status)● Dependencies (Start DB before App)● Logfiles (Rotate, Remove)● Disk Space● Monitoring and Alarming● ...
OPS learn from DEV to think about:
● Incremental Improvement● Infrastructure as Code● Version Control System● Coding (OO, Functions, Libraries …)● Code Quality● Unit & Integration Tests● Test Automation● ...
Untested=
Broken
Unit TestsTest the smallest possible components in an artifical environment.
System TestsTest the entire application in a real(istic) environment together with other applications.
Part of build process
Syntax checks
Scripts
Config Files
Data Files
Unit tests for functions/libs
Run program with test data
Check result
Check program behaviour with wrong/broken test data
Also run on Developer desktop
Quick feedback (~ seconds)
Install on test server
Run tests from outside
HTTP calls
Send emails
Try to login
Run tests from inside
Remote Exec (rsh, ssh …)http://go.schapiro.org/rshpitfall
Service Start, Stop & Status
Modify server to create good & bad test scenarios
Reboot
Unit Tests System Tests
UnitTests
%prep%setup -q
%installinstall … %{buildroot}/…install … %{buildroot}/…
%files%defattr(-,root,root,-)/...
BuildRequires: sudo%buildset -evisudo -c -f sudoers %installinstall -m 0440 sudoers -D \ %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-)/etc/suoders.d/%{name}
BuildRequires: PyYAML, pylint%buildset -e# syntax checksbash -n my_script.sh# Should be valid python codepylint -E yum-repo-propagate# should be valid YAML filepython -c "↩import yaml ↩yaml.safe_load(open('config.yaml'))↩"...
SystemTests
http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg
BuildAuto-
mationSourceCode(SVN)
Monitor
Changes
Build Server
Run BuildJob
● Check out source● Run Unit Tests● Create RPM●
Sourc
eCod
eDEVYUMRepo
UploadPROYUMRepo
PropagateRPM
Test Server
Deploy
and
Run
Test
Job
yum
Prod Server
Deploy
to PROD
yum
BuildAuto-
mationSourceCode(SVN)
Monitor
Changes
Build Server
Run BuildJob
● Check out source● Run Unit Tests● Create RPM●
Sou
rce
Cod
eDEVYUMRepo
UploadPROYUMRepo
PropagateRPM
Test Server
Deploy
and
Run
Test Job
yum
Prod Server
Deploy
to PROD
yum
SAN mount service
Test via rsh
Mock SAN deviceswith losetup
service start, stop mounts/umounts
Error handling
HTTP Proxy Configuration
X-Forwarded-For header spoofs source
Check result for ERR_ACCESS_DENIED
Run tests for all function groups!
Test Server
GET http://external.com/X-Forwarded-For: 10.11.12.01
502 Bad Gateway✔GET http://external.com/X-Forwarded-For: 10.34.56.01
403 Forbidden
ERR_ACCESS_DENIED by proxy server
✘✘
VM Provisioning & Kickstart Installation
Test via HTTP API
Create broken VMs and check error reports
Create valid VM and install Linux OS
Scrape VM screen via OCR
http://github.com/Immobilienscout24/lab-manager-light
34 35 36
2 3
53
87 88 89 90 91 92 93 94 95 96 97
TIME
Continous Live Deployment
Deploy every application when it is ready.Automate the delivery chain from source till production.
Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin
Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de
Thank you very much!Please contact me for further questions and discussions.
Top Related