Nice and Secure: Good OpSec Hygiene with Puppet!
Peter SouterProfessional Services Engineer | Puppet
@petersouter
@petersouter 2
Who am I?
@petersouter
Professional Services Engineer
5 years using Puppet
2 years @ Puppet Inc
Help customers deploy Puppet Enterprise
Teach Puppet classes
petems IRC/Slack/GitHub
@petersouter
Warning: I speak quicklyAnd I have a different accent...
3
@petersouter
My feelings on Q&Ahttp://bit.ly/why_no_talk_qa
● Tweet me @petersouter● Come up after this talk ● Meet me in the hallway
4
When will this QA be over so I can
leave?
We’ve got lots to cover - No Q&A!
@petersouter
So, why are we here?(This room specifically, listening to this talk...)
5
@petersouter 6
Every time someone uses this picture, Pete Cheslock gets his wings!
https://twitter.com/petecheslock/status/595617204273618944
@petersouter
Show of hands in the roomLet’s take the temperature of security here
7
@petersouter
Why is Puppet good for security?
Infrastructure as code RBAC Auditing Enforcement
8
@petersouter
Don’t let Puppet be the attack vector!aka. How do we make sure we’re not pushing the problem elsewhere?
9
@petersouter
What is OPSEC?Before we talk about something we should define it
10
@petersouter
“Operations Security, or OPSEC, is the process by which we protect unclassified information that can be used against us. OPSEC challenges us to look at ourselves through the eyes of an adversary (individuals, groups, countries, organizations). Essentially, anyone who can harm people, resources, or mission is an adversary.”
11
Department of Defense Education Activityhttp://www.dodea.edu/offices/safety/opsec.cfm
@petersouter 12
● Keeping your code clear of sensitive information
● Approaches to secrets management with the Puppet toolchain
● Making sure security is part of your workflow, rather than an afterthought
What are we going to cover?
https://flic.kr/p/7LcF2W
@petersouter
Let’s start with secrets...
13
We’ve all got them...
@petersouter 14
What are secrets in IT?
RadioactiveConsequences are dire from a leak
ExamplesPasswords, API Keys, SSH Keys, SSL Certs...
SmallA few kb at most
RequiredThe infrastructure won't work without them!
https://flic.kr/p/dHrwpb
@petersouter
Easiest to hardest● Avoid exposing secrets in logs
● Remove data from code and into the data layer (hiera)
● Encryption
15
How do we avoid exposing secrets in Puppet?
https://flic.kr/p/aCJZrf
@petersouter
Don’t expose secrets in logsKeep your secrets hidden
16
@petersouter
show_diffThe first place for leaks
17
@petersouter 18
root@homebox:~# puppet agent --show_diff
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content:
--- /etc/sensitive 2016-08-14 23:01:37.036863915 +0100
+++ /tmp/puppet-file20160814-24654-ak1ywd 2016-08-14 23:01:56.852882307 +0100
@@ -1 +1 @@
-Not Secret
\ No newline at end of file
+SECRET-CONTENT
\ No newline at end of file
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed
'{md5}2ab96390c7dbe3439de74d0c9b0b1767' to '{md5}44c7be48226ebad5dca8216674cad62b'
Notice: Applied catalog in 0.20 seconds
How it looks...
@petersouter
Anywhere reports go:
● syslog● interactive terminal output● PE Console● ENC● report processors
19
Where does the information from show_diff go?
@petersouter 20
file { ‘/etc/secrets.txt’:
ensure => 'file',
owner => 'root',
mode => '0600',
content => 'hunter2',
show_diff => false,
}
Setting show_diff to false at the resource level
@petersouter 21
An example from a Supported Module: mysql
file { "${::root_home}/.my.cnf":
content => template('mysql/my.cnf.pass.erb'),
owner => 'root',
mode => '0600',
}
# show_diff was added with puppet 3.0
if versioncmp($::puppetversion, '3.0') >= 0 {
File["${::root_home}/.my.cnf"] { show_diff => false }
}
https://github.com/puppetlabs/puppetlabs-mysql/blob/d58a100fa67bc99b4388d4ea3921b11647d483d7/manifests/server/root_password.pp#L39
@petersouter
Setting show_diff to false at resource scopeshow_diff = false
22
root@homebox:~# puppet apply secret.pp
Notice: Compiled catalog for homebox.home in environment production in 0.10 seconds
Notice: /Stage[main]/Main/File[/etc/sensitive]/content: content changed '{md5}d3b07384d113edec49eaa6238ad5ff00' to
'{md5}44c7be48226ebad5dca8216674cad62b'
Notice: Applied catalog in 0.19 seconds
@petersouter
There’s a balance...Hiding diffs reduces visibility of change...
23
@petersouter
inifile module to hide changesAllows you to only hide the sensitive fields
24
@petersouter 25
ini_file now has show_diff from the 1.5.0 release
@petersouter
Setting show_diff on individual sensitive fieldsshow_diff = false
26
ini_setting { 'ACME App Timezone':
section => 'TimeDate',
setting => 'TimeZone',
value => $acme_app_time_zone,
}
ini_setting { 'ACME App Password:
section => 'Settings',
setting => 'Password',
value => $acme_app_password,
show_diff => false,
}
@petersouter
Sensitive typeNew for the Puppet 4.6+ release
27
@petersouter 28
file { '/etc/sensitive':
ensure => 'present',
owner => 'root',
group => 'root',
content => Sensitive('hunter2'),
}
root@homebox:~# puppet apply secret.ppNotice: /Stage[main]/Main/File[/etc/sensitive]/ensure: changed [redacted] to [redacted]Notice: Applied catalog in 0.18 seconds
Ability to redact strings with the new Sensitive Type
@petersouter 29
$secret = Sensitive(‘Unwrapped’)
$unwrapped = $secret.unwrap |$sensitive| { $sensitive }
notice("Unwrapped: ${unwrapped}")
$secret.unwrap |$sensitive| { notice("Lambda: ${sensitive}") }
Unwrapping the secrets
https://www.devco.net/archives/2016/09/05/puppet-4-sensitive-data-types.php
@petersouter 30
You can use a dedicated redacted resource
Still on < 4.6?
30
@petersouter 31https://github.com/openstack/puppet-barbican/blob/2e2b10ae58fdc9ad27d88d3195260ef02af853ad/lib/puppet/type/barbican_config.rb
newproperty(:value, :array_matching => :all) do
desc 'The value of the setting to be defined.'
munge do |value|
value = value.to_s.strip
value.capitalize! if value =~ /^(true|false)$/i
value
end
newvalues(/^[\S ]*$/)
def is_to_s( currentvalue )
if resource.secret?
return '[old secret redacted]'
else
return currentvalue
end
end
def should_to_s( newvalue )
if resource.secret?
return '[new secret redacted]'
else
return newvalue
end
end
@petersouter 3232
Encrypt secrets on a node by node basis
Try binford2k-node_encrypt
32
@petersouter 33https://github.com/binford2k/binford2k-node_encrypt
● Master encrypts secrets for each node using their own certificate● Secret can only be decrypted with the node's private key● Uses built-in Puppet CA, so the base case is zero-config
node_encrypt::file {'/etc/company_app/credentials':
ensure => file,
owner => 'root',
content => 'hunter2', # transparently encrypted
}
How does it work?
@petersouter
node_encrypt
34
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109738'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/ensure: created
Notice: Applied catalog in 9.33 seconds
$ echo blah > /tmp/foo
$ puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for master.puppetlabs.vm
Info: Applying configuration version '1450109821'
Notice: /Stage[main]/Main/Node[default]/Node_encrypt::File[/tmp/foo]/Node_encrypted_file[/tmp/foo]/content: content changed '<<encrypted>>' to
'<<encrypted>>'
Notice: Applied catalog in 7.61 seconds
@petersouter
So data is no longer exposed in logsBut the data is still visible in the code!
35
@petersouter
Remove data from codeEspecially organisation specific data
36
@petersouter
Bad!Don’t do this...
37
@petersouter 38
class example_company_app {
if $::fqdn == 'prod.example.com' {
class {'company_app':
ensure => 'present',
password => 'hunter2',
ssl_enable => true,
}
} else {
class {'company_app':
ensure => 'present',
password => 'example123',
ssl_enable => false,
}
}
}
@petersouter
Good!Do this...
39
@petersouter 40
class profile::example_company_app {
$app_password = hiera('profile::example_company_app::password')
$app_ssl_enable_password = hiera('profile::example_company_app::password')
class {'company_app':
ensure => 'present',
password => $app_password,
ssl_enable => $ssl_enable,
}
}
@petersouter
Roles and profiles help a lotAbstracting implementation specifics away
41
@petersouter
Allows you to set organisational defaults in your roles and profiles
42
● Keep organisational specific data in hiera● Move organisational specific setup into role and profile wrappers
Advantage: Not only more secure, cleaner code that’s more reusable!
@petersouter
Default parameters are important!Abstracting implementation specifics away
43
@petersouter
Storytime: OpenstackThe perils of bad defaults...
44
@petersouter 45https://archive.fosdem.org/2015/schedule/event/public_puppet/
The open source OpenStack project infrastructure
Fully public Puppet
@petersouter
The more abstracted your control-repo, the less chance of leaking or compromising of secrets...
46
@petersouter
Another gotcha: hierarchy lookupsAka. Why trusted facts are good!
47
@petersouter
Take a look at this hiera config...
48
# hiera.yaml
---
:hierarchy:
- "node/%{fqdn}"
- "common"
:backends:
- yaml
:datadir: '/etc/puppet/environments/%{environment}/hieradata'
@petersouter
Facts are spoofable!
49
[root@testbox]# facter fqdn
pe-201620-master.puppetdebug.vlan
[root@testbox]# FACTER_fqdn=evil.example.com facter fqdn
evil.example.com
Facts are spoofable
@petersouter
Trusted facts got your back!
50
Locked in from the certificate request
@petersouter
Trusted facts are stamped on Node creation
51
Trusted facts are stamped on Node creation
@petersouter
We have a bunch of OIDs for this also...
52https://docs.puppet.com/puppet/latest/reference/ssl_attributes_extensions.html
@petersouter
New, better hiera hierarchy...
53
# hiera.yaml
---
:hierarchy:
- "node/%{trusted.certname}"
- "common"
:backends:
- yaml
:datadir: '/etc/puppet/environments/%{environment}/hieradata'
@petersouter
Theoretically, you should be able to release most of the code you write publically
without any sort of security issues
54
@petersouter 55
This is actually a tenet of 12 Factor Apps...
Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.
A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.
Note that this definition of “config” does not include internal application config, such as config/routes.rb in Rails, or how code modules are connected in Spring. This type of config does not vary between deploys, and so is best done in the code.
http://12factor.net/config
@petersouter
Example: GDSGovernment Digital Service, UK
56
@petersouter 57
Meeting the Digital Service Standard
To meet point 8 (understand security and privacy issues) you must:
● Make all new source code open and reusable
● Publish code under an appropriate licence
● Explain your reasoning for any code you haven’t made open
You’ll have to explain how you did this at your service assessments.
https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable
@petersouter 58
Meeting the Digital Service StandardWhen GOV.UK was first set up we were unable to publish our Puppet repository because our code and secrets were tied together. This goes against patterns like the 12-factor app which “requires strict separation of config from code”
This wasn’t true for our Puppet repository, but we gradually moved our credentials into a separate repository (rotating them as we did so).
“A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.”
@petersouter 59
$ strings modules/**/*.pp | tr ' '
'\n' | sort -n | uniq | view -
Check code for unique strings that look secret-y
Note: Requires zsh for the strings function!
@petersouter
It’s not just Puppet code!Git commits can sustain sensitive data!
60
@petersouter 61
$ git commit -a -m "Changed the
password to password1"
@petersouter
$ while read line; do echo $line;
git --no-pager log -p -S $line; done
< puppet_search
62
Manually searching through git commits for sensitive information...
@petersouter
Opening GOV.UK’s Puppet Repository https://gdstechnology.blog.gov.uk/2016/01/19/opening-gov-uks-puppet-repository/
Git Repo https://github.com/alphagov/govuk-puppet
Want to know more?
63
@petersouter
Your data is now separated. Hooray!But it’s still plaintext in Hiera. Boo! :-(
64
@petersouter
EncryptionTold you we’d come back to it!
65
@petersouter
Bad!Don’t do this!
66
@petersouter 67
@petersouter
Good!Do this!
68
@petersouter 69
Preso title goes here. To update, go to File > Page Setup > Header/Footer, paste title, Apply All
@petersouter 70https://github.com/TomPoulton/hiera-eyaml
Hiera eyaml
@petersouter
hiera-eyaml is probably the best method for internal data encryption with PuppetIt’s widely used, and has a number of plugins
71
@petersouter
eyaml plugins
72
● https://github.com/sihil/hiera-eyaml-gpg● https://github.com/tehmaze/hiera-eyaml-secretbox● https://github.com/acidprime/hiera-eyaml-pkcs11● https://github.com/adenot/hiera-eyaml-kms● https://github.com/gtmtechltd/hiera-eyaml-twofac
@petersouter
The idea is that Puppet will natively support encrypted data in the future
Follow this ticket for the roadmap view: PUP-1974
73
@petersouter
● TranscryptGit-CryptBlackbox
● Turtles All The Way Down: Storing Secrets in the Cloud and in the Data Center behind Closed Doors
VCS based encryption
74http://danielsomerfield.github.io/turtles
https://www.youtube.com/watch?v=OUSvv2maMYI
@petersouter
Dedicated secret devicesGoing deeper...
75
@petersouter
Why use a secret server?
76
● Dynamic secrets● ACL (Access control policies)● Leasing and renewal● Revocation● Encryption● Auditing● Supportability
@petersouter 77
Conjur, Vault, Keywhiz, Amazon KMS, Confidant
@petersouter 78
Hiera will plug into any secret service app with a little bit of Ruby glue
Hiera is just key/value lookup A hiera backend to basically any secret server setup is possible
@petersouter 79
$planet = conjur_variable('planet')
file { '/etc/hello.txt':
content => "Hello ${planet}!\n"
}
conjurize_file { '/etc/hello.txt':
variable_map => {
planet => ‘!var puppetdemo/planet’
}
}
https://www.conjur.net/puppet-secret-serverhttps://forge.puppet.com/conjur/conjur
Conjur and Puppet
@petersouter 80https://github.com/jsok/hiera-vault
hiera-vault
@petersouter
If you want to know more about Puppet + Vault, Seth Vargo from Hashicorp is presenting tomorrow
81
@petersouter
Behind Closed Doors - Managing Passwords in a Dangerous World by Noah Kantrowitz
● Really great in-depth presentation
● https://coderanger.net/talks/secrets/
● https://www.youtube.com/watch?v=TVEfYO-5-RA
● Great breakdown of secret management, advantages and disadvantages of approaches and tooling
Want to know more about secrets?
82
@petersouter
Cleaning up the current codebase
83
How to find secrets currently exposed
@petersouter
Manual Grepping
84
$ git grep -i -e
"(api\\|key\\|username\\|user\\|pw\\|password\\|pass\\|email\\|mail
)" -- `git ls-files | grep -v .html` | cat
@petersouter
--------------------------------------------------------------------
gittyleaks' Bot Detective at work ...
--------------------------------------------------------------------
file: site/profiles/templates/rhn/RHN-ORG-TRUSTED-SSL-CERT.erb
what: Key
value: (2048
match:
Public-Key: (2048 bit)
num_of_revisions: 59
Gittyleaks
85https://github.com/kootenpv/gittyleaks
@petersouter
Scumblr
86https://github.com/Netflix/Scumblr
@petersouter
Unfortunately, there’s no silver bullet to detect leaked credentials or proper usage of encryption...
87
@petersouter
A lot of it is about process, gating and reviews
88
@petersouter 89
Sometimes the job is too big for a repository, and it’s better to migrate to a cleaner repo
@petersouter
Ensuring it stays clean
90
After cleanup, ensuring it stays clean
@petersouter 91
At a minimum, you want to make sure that what you’re doing with Puppet isn’t
making things worse!
@petersouter 92
It’s largely a people and process problem
@petersouter
Making sure security is part of your workflow, rather than an afterthought“Shift security left”
93
@petersouter 94
“To keep up with the pace of Continuous Delivery, security must “shift left,” earlier into design and coding and into the automated test cycles, instead of waiting until the system is designed and built and then trying to fit some security checks just before release. In DevOps, security must fit into the way that engineers think and work: more iterative and incremental, and automated in ways that are efficient, repeatable, and easy to use.”
- DevOpsSec: Delivering Secure Software Through Continuous Delivery, Jim Bird
Shifting left!
@petersouter 95
How do we check things aren’t getting worse?
● Game days and internal evil attempt teams● Continuous security integration (Gittyleaks/code-review)● Dedicated security stories for sprints
○ Evil users or (mis)use cases● Embedded security team members● Dedicated audits on sensitive apps/stacks from external firms
@petersouter 96
Game Day example: Agent spoofingLet's say someone gets access to an agent.
What’s the worst they can do?
@petersouter
As soon as security becomes a blocker, you’ve lost!Security has to be automated where possible, otherwise we’re back to the throw-over-the-wall problems of pre-DevOps!
97
@petersouter
SummaryWhat have we learnt?
98
@petersouter
Remove sensitive data from your logsUse the new sensitive type or write custom providers
99
@petersouter
Use the roles and profiles pattern for original defaultsSeparate secrets, reduce the surface area for problematic setups and always read the docs before using Puppet modules
100
@petersouter
Use Trusted Facts in your hiera hierarchyMake sure hiera lookups can’t spoofed
101
@petersouter
Encrypt the separated dataHiera-eyaml or a dedicated secret server
102
@petersouter
Ensure your code stays cleanPeople, processes and automated testing
103
@petersouter
Move security leftMake it a part of your process, rather than an afterthought
104
@petersouter
Top Related