PuppetConf 2016: The Future of Testing Puppet Code – Gareth Rushgrove, Puppet
-
Upload
puppet -
Category
Technology
-
view
120 -
download
0
Transcript of PuppetConf 2016: The Future of Testing Puppet Code – Gareth Rushgrove, Puppet
(without introducing more risk)
The Future of Testing Puppet Code
PuppetGareth Rushgrove
Who, What, Why, How
(without introducing more risk)
@garethr
(without introducing more risk)
Gareth Rushgrove
(without introducing more risk)This TalkWhat we’ll cover
- Why test Puppet anyway- Existing Puppet testing tools- Different users and usecases- Future challenges
Gareth Rushgrove
Mainly a talk about problems,rather than solutions
Gareth Rushgrove
Not a talk about how to testPuppet code. Lot of existing content on that subject
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
PuppetConf 2014
(without introducing more risk)
Gareth Rushgrove
Survey of 300 Puppet users
(without introducing more risk)
Gareth Rushgrove
(without introducing more risk)Why Test Puppet
And a little history
Gareth Rushgrove
- You have lots of Puppet code- You’re targeting a wide range of OS’s- The Puppet language has more features
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
Lots of codeBut how much is lots
Note that this just represents public code on GitHub. Lots of private Puppet code out there too
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
How many .pp files on GitHub?
SELECT COUNT(*) FROM [puppet.puppet_files]
-- 489725
(without introducing more risk)
Gareth Rushgrove
How many lines of Puppet code?
SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line FROM [puppet.puppet_content] )
-- 6697099
(without introducing more risk)
Gareth Rushgrove
How big are .pp files?
SELECT QUANTILES(total_lines, 10) AS q FROM ( SELECT COUNT(line) total_lines FROM ( SELECT SPLIT(content, '\n') AS line, id FROM [puppet.puppet_content] ) GROUP BY id )
50% of Puppet files are 20 lines or less. 90% of Puppet files are 100 lines or less
Gareth Rushgrove
The largest Puppet file public on GitHub has 25,238 lines
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
Mixed environmentsSo many platforms
Gareth Rushgrove
A third of respondents run more than one OS on there development machine
Gareth Rushgrove
Gareth Rushgrove
60% of respondents run more than one OS in production
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
The Puppet language
Types, lambdas, iterators, functions
Gareth Rushgrove
(without introducing more risk)Existing Tools
Providing some context
Gareth Rushgrove
(without introducing more risk)Core tools
Common starting points
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
puppet-lint
(without introducing more risk)
Gareth Rushgrove
Style conformance
$ puppet-lint /etc/puppet/modules foo/manifests/bar.pp - ERROR: trailing whitespace found on line 1 apache/manifests/server.pp - WARNING: variable not enclosed in {} on line 56 ...
(without introducing more risk)
Gareth Rushgrove
rspec-puppet
(without introducing more risk)
Gareth Rushgrove
Unit tests for Puppet
it { is_expected.to compile } it { is_expected.to contain_class('db') } it do is_expected.to contain_package('mysql-server') .with_ensure('present') end it { is_expected.to have_exec_resource_count(0) }
(without introducing more risk)
Gareth Rushgrove
puppet-syntax
(without introducing more risk)
Gareth Rushgrove
Popular Puppet gemsSELECT package, COUNT(*) count FROM ( SELECT REGEXP_EXTRACT(line, r'gem [\'|"](.*)[\'|"]') package, id FROM ( SELECT SPLIT(content, '\n') line, id FROM [puppet.gemfile_contents] WHERE content CONTAINS 'gem' ) GROUP BY package, id ) GROUP BY 1 ORDER BY count DESC LIMIT 30;
(without introducing more risk)
Gareth Rushgrove
Popular gem results
puppetlabs_spec_helper 1157 puppet 1142 rake 1135 puppet-lint 768 beaker-rspec 610 rspec-puppet 576 metadata-json-lint 566 beaker 440 puppet-blacksmith 425 serverspec 359
(without introducing more risk)
Gareth Rushgrove
puppetlabs_spec_helper
(without introducing more risk)
Gareth Rushgrove
Advanced toolsNext steps
(without introducing more risk)
Gareth Rushgrove
beaker-rspec
(without introducing more risk)
Gareth Rushgrove
Serverspec
(without introducing more risk)
Gareth Rushgrove
rubocop
(without introducing more risk)
Gareth Rushgrove
Other toolsAlternative approaches
(without introducing more risk)
Gareth Rushgrove
puppet-catalog-test
(without introducing more risk)
Gareth Rushgrove
Test catalogue compilation
$ puppet-catalog-test -m modules -M site.pp [INFO] Using puppet 3.0.2 [PASSED] foo (compile time: 0.168182 seconds) [PASSED] default (compile time: 0.003451 seconds)
---------------------------------------- Compiled 2 catalogs in 0.1717 seconds (avg: 0.0858 seconds)
(without introducing more risk)
Gareth Rushgrove
Test Kitchen
(without introducing more risk)Different Users
And different usecases
Gareth Rushgrove
Half of respondents regularly writetests for Puppet code
Gareth Rushgrove
A quarter of respondents never write tests for Puppet code
Gareth Rushgrove
The problem with a tool-centricworld view
Gareth Rushgrove
Understanding user needs meansstarting with users
Gareth Rushgrove
Gareth Rushgrove
- The Vox Pupuli folks- Lots of Puppet employees- You in-house Puppet expert- Custom functions
The module authors
The regular Puppet users
Gareth Rushgrove
- Roles and profiles- Using Puppet weekly or more- PuppetDB, Puppet Server- Custom facts
The occasional Puppet user
Gareth Rushgrove
- Using Puppet as needed- Basic Puppet code- Hieradata
Today different types of Puppet users have a very different experience
Gareth Rushgrove
Gareth Rushgrove
For those with no Ruby experience the average was 2 out of 5
Gareth Rushgrove
For those with a little Ruby experience the average was still only 2.5 out of 5
Gareth Rushgrove
Of the people who love the testing experience 33% are self-described power users of both Puppet and Ruby
Gareth Rushgrove
Gareth Rushgrove
But more than 50% of respondents have little or no Ruby experience
Gareth Rushgrove
(without introducing more risk)Challenges
What we need for the future
(without introducing more risk)
Gareth Rushgrove
Improved getting started experience
Helping a new generation of users
- Getting started testing documentation- Single entry point for users- Opinionated workflow
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
Adopting acceptance testing
Still too hard
27 of the 67 Serverspec users are not using either Beaker or Test Kitchen.
Gareth Rushgrove
Only 5 of the Beaker (without rspec) users are not also using beaker-rspec
Gareth Rushgrove
3 of the Test Kitchen users are also using some flavour of Beaker
Gareth Rushgrove
So only 26% to 35% of respondents are using an acceptance testing tool with there Puppet code
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
Experience mattersConsistence interfaces
(without introducing more risk)
Gareth Rushgrove
Inconsistencies
$ puppet-lint vs rake lint vs rake validate $ rake syntax vs puppet parser validate $ rspec vs rake spec $ BEAKER_destroy=no rake beaker vs beaker
We have a language style guide. What about other Puppet user interfaces?
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
Ruby heritageRuby as a development environment
60% of respondents described themselves as Puppet power users
Gareth Rushgrove
Only 12% of respondents described themselves as Ruby power users. A third of those work for Puppet!
Gareth Rushgrove
- More help with using Ruby for testing?- Support for testing in other languages?- Native Puppet support for testing?- Containerised testing abstraction?
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
jordan/spec
(without introducing more risk)
Gareth Rushgrove
Test Puppet with Puppet
file { '/tmp/test': ensure => present, content => 'The file content', }
assertion { 'that the file has the correct contents': subject => File['/tmp/test'], attribute => 'content', expectation => 'The file content', }
(without introducing more risk)Conclusions
Where to go from here
Thanks to the community we have a collection of powerful testing tools in the Puppet ecosystem
Gareth Rushgrove
But we don’t have consistent user interfaces or the opinionated workflows useful for beginners
Gareth Rushgrove
Testing shouldn’t just be apower user feature
Gareth Rushgrove
As a community we should talkabout design more often
Gareth Rushgrove
(without introducing more risk)
Gareth Rushgrove
https://goo.gl/forms/nchOp5hYdLLgmJ8z1
Gareth Rushgrove
(without introducing more risk)Questions?And thanks for listening