Building A Self-Documenting Application: A Study in Chef and Compliance

2015 #ChefConf WorkshopBuilding a Self-Documenting Compliant Application

www.conjur.net | @conjurinc

© Confidential

Set-up Prerequisites

1.Sign up for the Conjur demo appliancehttp://info.conjur.net/secrets-management-demo

2. Login to the demo applianceIf you’re on Windows, use Putty

http://bit.ly/1NAHh3Y

Ready To Deep Dive?

http://info.conjur.net/secrets-management-demo

http://info.conjur.net/secrets-management-demo

http://www.putty.org/



© Confidential

Welcome!

I. AgendaII. Workshop Goals

III. Meet Your Workshop Hosts, Conjur IntroductionIV. Ready To Deep Dive? Set-up Conjur Prerequisites

● Part I: Introduction to DevOps and Compliance / Q&A ● Part II: Chef Secrets Management Overview / Q&A

Agenda

© Confidential

- Break -

● Part III: Practicum - Build a Sample App● Part IV: Compliance & Chef; Good Practice / Q&A ● Part V: Traceability, Auditability and Chef / Q&A

V. Wrap up

Agenda

© Confidential

Workshop GoalsAfter this workshop, you will:

● Have a deeper understanding of how compliance with Chef, access to the Chef Server, and Secrets Management are interrelated.

● Know how to implement a delivery pipeline that combines the two topics in a self-documenting and highly automated fashion.

© Confidential

Your Workshop Hosts

● Kevin Gilpin, CTO and Co-Founder, Conjur - Kevin has 19 years of technical leadership in

enterprise software, delivering solutions to the Fortune 500. He has held technical positions at Infinity

Pharmaceuticals, Praxeon Inc, i2 Technologies, and Trilogy, where he developed technology that

delivered pioneering interactivity on the web and has ultimately been adopted across the industry.

@kegilpin

● Elizabeth Lawler, CEO & Co-Founder, Conjur - Elizabeth Lawler is CEO and Co-founder of

Conjur, Inc. Elizabeth has over 20 years of experience working in highly regulated and sensitive data

environments. Prior to founding Conjur, Elizabeth was Chief Data Officer of Generation Health and held

a leadership position in research at the Department of Veterans Affairs. She has been a programmer

herself and is constantly working to make software development and IT systems easier to manage for

people working in regulated industries. @elizabethlawler

© Confidential

Your Workshop Hosts

● Dustin Collins, Developer Advocate, Conjur - Dustin is a polyglot engineer and Conjur's

developer advocate. He organizes the Boston DevOps meetup and is most interested in how we as a

community can iterate on our processes and tools to enable continuous delivery of quality software.

@dustinmm80

● Christopher Farnham, Novartis - Chris is a proven software development and operations team

leader within diverse domains, including banking, publishing, military and biotechnology. He has

demonstrated deep and broad knowledge of SDLC processes within highly regulated, demanding (but

ultimately person driven) environments. @chrisfarnham

https://twitter.com/chrisfarnham

© Confidential

Conjur IntroductionWe believe that security success for all organizations is

determined at the business and infrastructure levels

● Conjur is a privileged access security platform for large and fast-growing cloud and web-based enterprises using next generation technologies.

● It provides a smarter, transparent, automated way to scale and manage critical secrets and access control. It’s used on-premise or in your cloud (private, public, hybrid).

© Confidential

Conjur Introduction

Application Auth

Systems Access

Internal Network

Physical Infrastructure

Firewall

Control Plane

We provide the modern access management and audit solution for the missing pieces: Systems Admin and the Systems Control Plane.

Conjur Resolves The Access Control Gap

© Confidential

Conjur Introduction● Conjur solves three critical infrastructure security growth

issues:

> Secrets Management> Access Control (SSH) (e.g., machines, robots, humans)> Audit (Compliance)

● Integration-Friendly! We work with all technology stacks.

© Confidential

Conjur Clients

© Confidential

Part I.

Introduction to DevOps and Compliance

Compliance for People who Cringe at “Compliance”

© Confidential

How the Business Views DevOps

How does DevOps work?

Magic.

© Confidential

Compliance and DevOps

Source: DevOps: The Worst-Kept Secret to Winning in the Application Economy CA Technologies October 2014 cite: http://rewrite.ca.com/us/~/media/rewrite/pdfs/white-papers/devops-winning-in-application-economy.pdf

© Confidential

Goal: Reproducible, Reliable Processes

● Same packages● Same configuration

… and…

● Same security and compliance controls

© Confidential

● Automated and traceable authorizations for promotion of code to production

● Role-based access controls that acknowledge when DevOps personnel have access to production systems and document the specific use cases

● Encryption and logical access controls which essentially “lock-out” unauthorized access (to systems or data)

Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-access-to-production/)

Auditors/Compliance want to see

https://www.brightline.com/2012/12/auditing-devops-developers-with-access-to-production/




© Confidential

● File integrity monitoring (and alerting) on changes to production code versus the traditional focus on critical operating system executable

● File access monitoring on the source code itself with appropriate alerting

● Extensive logging and daily, if not real-time, log review of the above data sources

Adapted From Brightline https://www.brightline.com/2012/12/auditing-devops-developers-with-access-to-production/)

Auditors/Compliance want to see





© Confidential

● Single source for all change requests

○ No change is deployed without a request

○ Intent is that most problems can be diagnosed via the change reporting systems

● Clear separation of roles between developer and deployer

○ Or at least developer cannot deploy without review

● Clear metrics to demonstrate improvement

○ Mean Time To Recovery

○ Percentage of failed deployments/changes

Goals for delivery automation

© Confidential

Dev, Test, & Prod Environments

Code Review

Infrastructure Source Code

Goal automate documentation of this

InfrastructureCode Developer

deploy

Continuous Build & Unit

Test

Config, Release, Deployment

commit on branch build

check

approval

tests pass

© Confidential

Part IIChef and Access Management

© Confidential

● Access control is determined by “gates” which can be opened and closed by human or automated processes

● SSH gates access to production machines by developers and code

● Secrets gate access to production services by developers and code

○ Access to a secret constitutes access to a production service

The Importance of SSH and Secrets

© Confidential

State of Chef + SSHCombining users + sudo cookbook is a popular solution.

Workflow

● Create a data bag for your users and place it in a my_users cookbook

● Create JSON entries for each user w/ group membership and public SSH keys

● Upload the data bag and cookbook to Chef serverInclude the sudo recipe in my_users cookbook to set passwordless sudo for your users

https://supermarket.chef.io/cookbooks/users

https://supermarket.chef.io/cookbooks/sudo

© Confidential

State of Chef + Secrets

chef-vault + IAM is a popular solution

workflow

● Create IAM JSON file defining access permissions to S3 buckets

● Script out creating IAM users/groups/policies for specific access to a set of secrets

● create a vault and encrypt the JSON output of the previous step, tied to a specific Chef query

http://sysadvent.blogspot.com/2013/12/day-19-automating-iam-credentials-with.html

http://sysadvent.blogspot.com/2013/12/day-19-automating-iam-credentials-with.html

© Confidential

State of Chef + Audit

Chef Analytics

● Tracks changes to Chef objects (clients, nodes, etc) and who made them in real-time

● You write controls into your recipes, compliance tests that report back to analytics after Chef run

Guardrail

● SAAS that tracks changes to infrastructure in real-time, can trigger alerts on specific changes

https://docs.chef.io/analytics.html

https://docs.chef.io/analytics.html

http://www.scriptrock.com/product

http://www.scriptrock.com/product

© Confidential

We will create a simple, but real infrastructure using Chef, Conjur, and Docker. Infrastructure activity and access will be fully managed and audited.

Workshop Flow - Overview

Chef Chef Server is the definitive source of server configuration.Chef deploy and validation keys provide access to Chef Server.Chef provides audit and verification of server installs.

Conjur Provides User, Group, and Host identity.Stores and distributes secrets.Manages SSH access to machines.Records audit of all permission changes, access to secrets, and access to hosts, plus custom audit records of cookbook pushes.

Docker Used to run infrastructure machines:deploy can push cookbooks to Chef Server. Accessible only to ops.sentry application server. Accessible to ops and developers.

© Confidential

◁ Move to the source directory and fetch all the code

Workshop Flow - Get the Code

cd chefconf15git submodule update --init --recursive

Submodule 'cookbooks/foundation' (https://github.com/conjurdemos/chefconf15-base.git) registered for path 'cookbooks/foundation'Submodule 'cookbooks/sentry' (https://github.com/conjurdemos/chef-sentry) registered for path 'cookbooks/sentry'…

© Confidential

◁ Create an ops group to own the key

◁ You’re a member of security_admin, so you’re also a member of ops.

Workshop Flow - Create the “ops” group

conjur group create --as-group security_admin ops

{ "id": "ops", "ownerid": "demo:group:security_admin", "gidnumber": null, "roleid": "demo:group:ops", "resource_identifier": "demo:group:ops"}

© Confidential

◁ Store the public key of the ‘demo’ user (that’s you) in Conjur.

Workshop Flow - Load your Public Key

conjur pubkeys add demo @~/.ssh/id_rsa.pub

Public key 'demo@ip-10-97-161-35' added

© Confidential

◁ Copy deploy.pem and validation.pem from your Chef Server

◁ Save to the local files : deploy.pem and validation.pem

◁ deploy.pem has permission to manage cookbooks

◁ validation.pem used to register new machines with Chef

Workshop Flow - Download the Chef Keys

© Confidential

Workshop Flow - Store the Chef deploy.pem

conjur variable create --as-group ops -m application/x-pem-file \chef-server/deploy/private_key

cat deploy.pem | conjur variable values add chef-server/deploy/private_key

{ "id": "chef-server/deploy/private_key", "mime_type": "application/x-pem-file", "kind": "secret", "version_count": 0}

Value added

◁ Create the variable and store the key

© Confidential

◁ Store the Chef server validation key in Conjur

Workshop Flow - Store the Chef validation.pem

conjur variable create --as-group ops -m application/x-pem-file \chef-server/validation/private_key

{ "id": "chef-server/validation/private_key", "mime_type": "application/x-pem-file", "kind": "secret", "version_count": 0}

cat validation.pem | conjur variable values add chef-server/validation/private_key

Value added

© Confidential

◁ Foundation image cookbook is pre-loaded into the Chef server. Along with Conjur auth and RBAC, this is the foundation of trust in the infrastructure.

◁ Base OS.

◁ Packages - Chef, Conjur

◁ SSH access - Conjur SSH - login with your own private key, always

◁ /etc/conjur.conf + /etc/conjur-demo.pem - Verified TLS for everything

◁ Only public info! No secrets.

https://github.com/conjurdemos/chefconf15-base

Workshop Flow - Foundation Image



© Confidential

“Deploy” server builds on the foundation image

◁ .conjurenv → Chef validation.pem

◁ client.rb → Chef client configuration

◁ deploy.sh → Docker entrypoint, uploads all cookbooks to Chef server. Cookbooks should be volume-mounted to Docker

Workshop Flow - Build Deploy Server

docker build -t deploy deploy/

© Confidential

Workshop Flow - Create Deploy Host

conjur host create --as-group ops deploy-01 | tee deploy-01.json

◁ Conjur - Create a Host identity for the Deploy server

◁ Assign ownership to the ops group

cat << ENV > deploy-01.envCONJUR_AUTHN_LOGIN=host/deploy-01CONJUR_AUTHN_API_KEY=$(cat deploy-01.json | jsonfield api_key)ENV

© Confidential

◁ Launch the deploy host

◁ Host does not have access to the secret … yet

Workshop Flow - Test the Deploy Host

docker run --env-file deploy-01.env --entrypoint conjur deploy env check

error: Some variables are not availableCLIENT_PEM: unavailable

© Confidential

◁ Give access to the Chef key to the deploy host

Workshop Flow - Grant Access to Chef

conjur resource permit variable:chef-server/deploy/private_key \host:deploy-01 \execute

Permission granted

© Confidential

◁ Run the deploy box as a daemon. We will use SSH to access it.

Workshop Flow - Launch the deploy box

docker run -d --env-file deploy-01.env \--name deploy \-v $PWD/cookbooks/:/cookbooks \-p 2200:22 \deploy

*** Running /etc/my_init.d/10_foundation-configure.sh...[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***…*** Runit started as PID 336Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up; version='3.5.3'

docker logs -f deploy

© Confidential

◁ Login to the deploy host

◁ On the deploy host, display user info. Group membership is controlled by Conjur.

Workshop Flow - Login to the deploy host

ssh -p 2200 localhost

demo@e9de200cc9da:~$

id

uid=1100(demo) gid=50000(conjurers) groups=50000(conjurers)

This icon means, run this command from inside a container

© Confidential

◁ Use the deploy host to list cookbooks

◁ Group ‘conjurers’ is whitelisted to passwordless-sudo this command.

Workshop Flow - List Cookbooks

sudo list-all

apt 2.7.0build-essential 2.2.1conjur-client 0.3.1foundation 0.1.1sshd-service 1.1.0terminal-login 0.2.3yum 3.5.3

© Confidential

◁ Exit the container with Control-D and navigate to the cookbooks directory

◁ Rename the sentry folder to end with your initials

Workshop Flow - Build the sentry Cookbook

cd cookbooks

mv sentry sentry-yourname ← changeme!cd sentry-yourname

© Confidential

Edit metadata.rb

◁ Change the cookbook name to end with your initials

◁ Increment the version to 0.2.0


© Confidential

◁ Edit recipes/default.rb


© Confidential

◁ Login to the deploy host

Workshop Flow - Login to the deploy host


demo@e9de200cc9da:~$

© Confidential

◁ This sequence will vendor the Berksfile in /cookbooks/sentry, then upload the cookbook with dependencies.

Workshop Flow - Upload Cookbooks

sudo upload sentry-yourname

# ./upload-all /cookbooks/sentry/Resolving cookbook dependencies...Fetching 'sentry-keg' from source at .Using build-essential (2.2.2)Using python (1.4.6)Using yum (3.5.3)Using sentry-keg (0.1.6) from source at .Using yum-epel (0.6.0)Vendoring build-essential (2.2.2) to /cookbooks/sentry//.vendor/build-essentialVendoring python (1.4.6) to /cookbooks/sentry//.vendor/pythonVendoring sentry-keg (0.1.6) to /cookbooks/sentry//.vendor/sentry-kegVendoring yum (3.5.3) to /cookbooks/sentry//.vendor/yumVendoring yum-epel (0.6.0) to /cookbooks/sentry//.vendor/yum-epelUploading sentry-keg [0.1.6]Uploaded all cookbooks.

© Confidential

◁ Audit activity shows access to secrets, as well as the upload event

Workshop Flow - Inspect the audit

conjur audit all -s

[2015-03-30 18:28:25 UTC] demo:host:deploy-01 checked that they can execute demo:variable:chef-server/deploy/private_key (true)[2015-03-30 18:28:30 UTC] demo:host:deploy-01 created resource demo:chef-cookbook:foundation owned by demo:host:deploy-01[2015-03-30 18:28:31 UTC] demo:host:deploy-01 reported cookbook-deploy:upload; message: Uploaded cookbook 'foundation' with command options: --include-dependencies

© Confidential

◁ Run a MySQL server

◁ Display the connection info

Workshop Flow - Launch database server

cd ~/chefconf15/databasesource start.sh

Launching databaseDatabase launched! Check $DB_HOST and $DB_PASSWORD

echo $DB_PASSWORD

eee5377d52c6b033

© Confidential

◁ Store the DB password in Conjur

Workshop Flow - Store the database password

conjur variable create --as-group ops \sentry/mysql/password \$DB_PASSWORD

{ "id": "sentry/mysql/password", "mime_type": "text/plain", "kind": "secret", "version_count": 1}

© Confidential

◁ Create the host

◁ Permit secrets access

Workshop Flow - Configure Sentry host

conjur host create --as-group ops sentry-01 | tee sentry-01.json

cat << ENV > sentry-01.envCONJUR_AUTHN_LOGIN=host/sentry-01CONJUR_AUTHN_API_KEY=$(cat sentry-01.json | jsonfield api_key)ENV

conjur resource permit variable:chef-server/validation/private_key \host:sentry-01 \execute

conjur resource permit variable:sentry/mysql/password \host:sentry-01 \execute

© Confidential

◁ Run the Sentry box as a daemon.

Workshop Flow - Run Sentry

docker run -d --env-file sentry-01.env \--name sentry \-e DB_HOST=$DB_HOST \-e COOKBOOK_NAME=sentry-keg \-p 2201:22 \-p 8080:9000 \sentry

*** Running /etc/my_init.d/10_foundation-configure.sh...[2015-03-28T20:56:58+00:00] INFO: Forking chef instance to converge...[2015-03-28T20:56:58+00:00] INFO: *** Chef 12.0.3 ***…*** Runit started as PID 336Mar 28 20:57:09 613420620e92 syslog-ng[345]: syslog-ng starting up; version='3.5.3'

docker logs -f sentry

Change this to match your cookbook name

© Confidential

◁ Login to the sentry box

◁ On the sentry box, run chef-client

Workshop Flow - Configure sentry


… chef-client run happens here

$ sudo conjur env run -c /.conjurenv -- sudo -E chef-client \-c /etc/chef/client.rb \-o $COOKBOOK_NAME

© Confidential

● Create a developers group and a user.

● Try to deploy, as this user.

Workshop Flow - Add New Team Members

© Confidential

DevOps Mission

Development Operations strives to shorten and amplify feedback loops, thereby bringing software development and IT operations closer to end users.

© Confidential

Macro Trends

● Self-service● Virtualization● Automation● Manage sets of assets not individual assets● External collaborations● Data Explosion● Globalizaton● Mobile devices● Internet of Things

© Confidential

In order for Chef to manage system configurations in our regulated environment it needed to be regulated itself. This is daunting; the following high-level strategy really helped me deliver

First: Describe the system and list all activities

Second: Go through each activity and ask:

◁ Can I declare this activity out of scope?◁ Can I reuse (or map it to) a pre-existing policy or activity?◁ Can I leverage automation (in lieu of documentation)?◁ Document the activity at an appropriate level

The above is generally ordered best to worst (but ultimately not that bad)

Chef in a regulated environment

© Confidential

◁ Gather product documentation

◁ List all components and integrations to the system

◁ List all activities with the system with careful attention to status updates

◁ Describe and list the roles and responsibilities associated with the system

Example activities:

◁ Create/Modify/Update Cookbooks in Chef Server

◁ Bootstrap (add)/Remove Node

Protip: Declare a “maintenance” activity with a change tracking process of “none” ; because nothing changes when this activity occurs

System activities

© Confidential

Always look to manage scope and be aware of which integrated systems are particularly sensitive (read: important) and which integrated systems are managed by other groups

◁ Declare which products and systems are “in scope”◁ Explicitly declare what is “out of scope”

Examples:

◁ Repository development activities (e.g., Berkshelf, cookbook writing) were declared “out of scope”

◁ Repository changes (e.g., upload, remove) were declared “in-scope”◁ Supporting infrastructure (e.g., monitoring, Jenkins) was declared “out of

scope”

Managing scope

© Confidential

Documentation and processes already existed for many things, we just reused them.

Examples:

◁ Server provisioning and retirement ← On-premise◁ Incident management (we just needed to declare roles and list points of

contact)◁ Backup and recovery

Reusing existing activities

© Confidential

Leveraging automationAutomated processes, where everything happens in a traceable and auditable

manner (driven out of assets managed in source control) provide great flexibility and insulation within the change control process.

Example doc excerpt: “change tickets must be filed for all medium risk updates unless performed via source control driven automation.”

© Confidential

Figuring out what should be documented at a high-level and what should be detailed is an art but important

Control related documentation is usually tracked and signed (a rigorous process). Do yourself a favor and only put core, less frequently changed information within the core doc (the operations manual). Then make heavy use of referenced “working instructions” that can be updated without going through a heavy process

Documenting activities

© Confidential

◁ Really get to know the compliance group

◁ Ask to attend meetings, request 1 on 1s, have lunch with them

◁ Goal is to learn their language, concerns and establish a personal relationship

◁ Learn the groups, communication paths, workflows and activities within the organization

◁ How are change management decisions made?

◁ Who are the operational and compliance stakeholders?

Homework

© Confidential

We hope you learned A LOT today!

Next Steps...> Test drive our on-demand demo at conjur.net > Check out our dev docs at developer.conjur.net> Set up a live 1:1 demo to learn more> Subscribe to our blog & monthly newsletter> Follow us @conjurinc

Thank You!

Building A Self-Documenting Application: A Study in Chef and Compliance

Software

Transcript of Building A Self-Documenting Application: A Study in Chef and Compliance