Recipe for Good Secrets Management

Boston Chef Meetup - August 2014

Secrets Management

A strength when done right

Otherwise…

◁ CodeSpaces

◁ AWS Credential leaks in Github

◁ HIPAA breach violations

◁ Heartbleed update woes

⊃ Was tricky even for the experts

⊃ Required BOTH openssl library update AND certificate rotation

First the bad news

“I don’t care about compliance”

Really? Would your company ever like to get

6-figure checks from large enterprises?

Secrets

Traditional approach

ConfigurationManagement(CM)

Artifacts

Orchestration

New Machine

“Secrets has always been kind of a hacky bit; like GPG encrypt a piece of data and stick it in a YAML file”

- Anonymous CM Technologist

Secrets Management is pretty hard

Requirement : separation of duties

One single actor should not be able to do everything.

Use separate roles for:

a) Loading credentials

b) Retrieving and using credentials

Requirement : least privilege

Only give each actor as much power

as is necessary to get the job done.

Requirement : leak-resistance

Don’t leave secrets lying around on:

◁ unencrypted persistent disks◁ backups◁ snapshots

Requirement : audit

Record changes:

◁ policies which govern access◁ each time a secret is changed◁ each time a secret is fetch and used

Requirement : rotation

A secret (e.g. database password, cloud credential)

Should be changed regularly

Here’s a recipe for good secrets management

can admin

can admin

can update

can update

can fetch

can fetch

“break glass”account

Environment A : “stage” Environment B : “production”

can admin

can admin

“break glass”account

Have an emergency access credential

Environment A : “stage” Environment B : “production”

Separate security environments

can update

can fetch

can fetch

Environment A : “stage” Environment B : “production”

Design for “robot” actors

Minimize human access to production

can update

can fetch

Environment B : “production”

can update

Environment B : “production”

Rotate credentials with privileged robots

Environment A : “stage” Environment B : “production”

Automatically record system activity

Where do Configuration Management

and Orchestration fit in?

Environment A : “stage” Environment B : “production”

Provisioning robots into the system

Bootstrapping machine identity

Orchestration Server

Launch Scriptor

Console

NewMachine

RobotIdentity Server

● New machine calls to Orchestration Server for identity

● Orchestration passes a credential (token) to Robot Identity Server

● Robot Identity assigns robot identity● Orchestration / CM installs identity on the new

machine

A new machine is impotent

until identity is acquired

Fetching secrets

password = secrets_manager.secret([node.

chef_environment,

'mysql/server_root_password'].join('/')

mysql_database 'phpapp' do

connection (host: 'localhost', username:

'root', password: password)

action :create

end

http://gettingstartedwithchef.com/first-steps-with-chef.html

● Replace sensitive attribute data with secrets from the secrets manager

● Use the environment name to separate secrets into permissions namespaces

template '/dev/shm/mysql.conf' do …endlink '/dev/shm/mysql.conf' do to '/etc/mysql.conf'end

Keep secrets separate from dataGeneral strategy for Linux - install secrets to /dev/shm

template '/mnt/etc/mysql.conf' do …endlink '/mnt/etc/mysql.conf' do to '/etc/mysql.conf'end

General strategy for ec2 - install secrets to /mnt

Keeping secrets separate from

data helps to satisfy important

compliance and security standards

such as PCI and HIPAA