About Me (Jason Denning)
• DevOps Engineer• SaltStack user for ~ 3 years• SaltStack Certified Engineer #2• AWS Certified Architect• OSS Advocate (first Linux install ~ 1998)• Pythonista• jasondenning on GitHub / @jason_denning on Twitter
● Provide analytics and marketing/segmentation tools for mobile app developers
● Hosted backend APIs available world-wide
● Lots of traffic● Big fans of SaltStack● upsight.com
WARNING: Work In Progress
● Testing infrastructure is not a common practice ● These slides are rough
● Your Mileage May Vary!
● We should work (together) on improving this
Why Test?
● DevOps: Infrastructure is code
● Code should be tested
● ⇒ Infrastructure should be tested
Why Test? (continued)
● Find errors faster
● Prevent regressions
● Allow others to contribute ● Simplify Code Reviews (CI)
● Improve Documentation
What to Test
● Basic Syntax (linting)
● Individual States
● State Relationships ● Different Environments
● As much as possible (within reason)
Testing Basics
● Tests should run quickly (and automatically)
● Tests should be isolated - know what you’re testing! ● Unit Tests: Test a single ‘unit’ of code
● Integration Tests: Test interactions between units
● Acceptance Tests: Test that code does what we want
Minimal Testing
$ salt-call state.highstate test=True
● Better than blindly applying states
● Still not what we’re looking for○ Manual○ Slow○ No isolation○ Error-prone (“Oops, I
didn’t see the red line!”)
First Improvement
Test states individually:
$ salt-call state.sls foo test=True
● Slight improvement○ More isolation○ Faster than highstate
● Still not what we want
Testing Basic Syntax of SLS Files
● First Attempt: Verify SLS files are valid YAML○ Problem: Only works for very basic states - Jinja breaks the test!○ Problem: What about other renderers?
● ● Improvement: Test that Salt can render the SLS file
○ “salt-call state.sls foo test=True” works for this○ Alternative: “salt-call state.show_sls foo”
Testing Basic Syntax
$ salt-call state.sls foo test=True(with syntax error)
$ salt-call state.show_sls foo(with syntax error)
Note: Better error message!
Testing Basic Syntax - Notes
● Salt built-ins are useful!
● Better: A separate tool for validating syntax (somebody should build one!)
● Don’t forget about outputters:
● salt-call state.show_sls foo --output=json
Testing in Isolation
● We need to be sure that test results are not influenced by previous tests
● Unit testing frameworks handle this automatically
● Ideal: Create a new, clean infrastructure for each test○ Problem: Tests must run quickly!
● Solutions:
○ Use VMs (still to slow if we’re running lots of tests)○ Use Containers (e.g. LXC / Docker)
Docker Containers vs. Full VMsDocker Pros:
● Easy● Quick to build and destroy
Docker Cons:● Missing Functionality (testing services will take
more work)● Linux only
VM Pros:● Full OS install - services work / any OS● Easy to manage if using tools such as Vagrant● Might be identical to your actual environment
VM Cons:● Slow(er) to build and destroy
Best Practice: Use both! (For different types of test)
Docker: Quickstart● Docker makes it easy to build & manage LXC containers● Allows one to quickly customize existing container images● Mount host directories in the container
Basic Dockerfile (~/saltpdx/Dockerfile):
FROM phusion/baseimageRUN sudo add-apt-repository -y ppa:saltstack/saltRUN sudo apt-get updateRUN sudo apt-get -y install salt-minionVOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]CMD salt-call state.highstate
● Build an image (tagged salt/test, version 1):$ docker build -t ‘salt/test:1’ .
● Start a container using the image:$ docker run --rm -it -v ~/saltpdx/salt:/etc/salt -v ~/saltpdx/states:/salt/states -v /
~/saltpdx/pillar:/salt/pillar salt/test:1
Docker Quickstart● Docker creates a container
based on the image
● Host directories are specified using the -v flag
● Docker starts the container and runs the CMD specified in the Dockerfile
● --rm flag tells docker to delete the container after the command has run
● -it (or -i -t) for interactive terminal mode (add ‘/bin/bash’ to end of command to get a shell)
Docker Tips
● Mount host directories as volumes for:○ Salt configuration○ States○ Pillar○ Logs?
● Configure containers as salt-masterless to avoid networking & PKI issues
● Use the --rm flag to automatically delete container instances after they’ve run
● Tag & version images
Docker: Entrypoint● When you run a Docker container, Docker executes a binary called ENTRYPOINT● The CMD that you pass to Docker (via Dockerfile or the cli) is passed as an argument to
the ENTRYPOINT● By default, Docker will use “/bin/sh -c” as the ENTRYPOINT - this can be overridden● e.g.:
$ docker run --rm --entrypoint /bin/echo salt/test:1 foo
● We can exploit this to make a simple test runner● Dockerfile:
FROM phusion/baseimageRUN sudo add-apt-repository -y ppa:saltstack/saltRUN sudo apt-get updateRUN sudo apt-get -y install salt-minionVOLUME ["/etc/salt", "/salt/states", "/salt/pillar"]ADD test_salt.sh /test_salt.shRUN chmod +x /test_salt.shENTRYPOINT ["/bin/bash", "/test_salt.sh"]
Docker: Entrypoint (con’t)After building a new image, we can run:$ docker run --rm -v / ~/saltpdx/salt:/etc/salt -v / ~/saltpdx/states:/salt/states -v / ~/saltpdx/pillar:/salt/pillar salt/test:2 / highstate
Which will start a container, run “salt-call state.highstate”, and exit.
Docker: Entrypoint (con’t)Or, we can run:$ docker run --rm -v / ~/saltpdx/salt:/etc/salt -v / ~/saltpdx/states:/salt/states -v / ~/saltpdx/pillar:/salt/pillar salt/test:2 / foo
Which will run “salt-call state.show_sls foo”, then “salt-call state.sls foo”, and exit.
Next Step: Automation
● We’ve got the basic tools - the next step is to add some automation● Host-side automation:
○ Starting and destroying containers○ Determine what states to test○ Collect & summarize test results○ Test various permutations (different Linux version, different
grains, etc.)
● Container-side automation○ Discover & test sub-components (e.g. file templates)○ Integration/Acceptance/Behavioral Tests
Automation Overview
● Testing is initiated by running a test-controller script on the host● The test-controller should:
○ Generate a list of states to test and environment permutations○ Start a container and trigger the test runner○ Collect test results○ Destroy the container○ Repeat for each state
● The Docker image should have a test-runner script that is used as the ENTRYPOINT
● The test-runner script should:○ Determine specific tests to run○ Run the tests○ Output Results
Next Time - Part 2
● Actual code for these automation scripts● Behavioral Testing (for acceptance/integration tests)● Collecting and Summarizing Results in a useful fashion● Continuous Integration● Adding Vagrant VMs for additional tests● Testing multi-machine infrastructures● …???
Top Related