Keeping Secrets: GKE with Hashicorp Vault Architecture Highlights 20 Dedicated GKE Cluster HashiCorp...
Transcript of Keeping Secrets: GKE with Hashicorp Vault Architecture Highlights 20 Dedicated GKE Cluster HashiCorp...
Secrets Overview
What is a “Secret”?
5
Username and Password
API Token
TLS Certificate
Credit Card
Tax File Number
Passport Number
Authorize and Authenticate
Confidential and sensitive
Secrets Overview
Secrets management
6
How do applications get secrets?
How do people get secrets?
How do the secrets get updated?
How do secrets get revoked?
When were secrets used and by whom?
What happens if a secret is compromised?
Secrets Overview
Where is the world now?
7
Secrets sprawled
Decentralized
Limited access control
Poor Visibility
Introducing Hashicorp Vault
So What does Vault do?
9
Single Source of Secrets
Programmatic Access (API)
Manual Access (CLI/UX)
Cloud Friendly
Introducing Hashicorp Vault
Secure Storage
10
Data encrypted in transit
Data encrypted at rest
Hierarchical key/value store
TLS 1.2 for clients
256 bit AES-GCM
$ vault write secret/foo Aaron_pw=Amiga1200Success! Data written to: secret/foo
$ vault read secret/fooKey Value---- -----Refresh_interval 768h0m0sAaron_pw Amiga1200
Introducing Hashicorp Vault
CLI Example
11
Writing and reading keys from Vault. Keys are stored in a hierarchical layout, and encryption is transparent to the client.
$ curl \--header ‘x-vault-token: …’ \--data ‘{“data”: {“Aaron_pw”: “Amiga1200”}}
\$VAULT_ADDR/v1/secret/data/foo
$ curl \--header ‘x-vault-token: …’ \$VAULT_ADDR/v1/secret/data/foo
Introducing Hashicorp Vault
API Example
12
Most interactions with Vault are also available via the API.
Vault Overview
13
What if ... ? An operator copies passwords to ~/passwords.txt
An application logs the database credentials to disk?
A server error includes the API token in the diagnostic output?
Vault Overview
14
Dynamic Secrets Dynamic secrets are ephemeral and unique
Unique credentials give us a tight audit trail
Ephemeral secrets time-bound access
Access can be revoked early in case of a breach
Vault Overview
16
Secrets Backends Cloud Providers
GoogleAWSAzure
Databases
(*)SQLOracleMongoDB
Other Systems
RabbitMQSSHTLS\PKIActive DirectoryHashiCorp NomadHashiCorp Consul
Vault Overview
17
Authentication
Vault clients provide their identity and authenticate before accessing secrets. Vault has platform specific integrations to asset client identity
Authorization
Client identity is mapped to groups and policies which govern which secrets are accessible. A fine-grained authorization language allows control of which resources and operations are available.
Auditing
Rich audit logging on requests and responses provides a full audit trail of which secrets were accessed by which clients.
Recommended Architecture on Kubernetes
19
Vault on GKE
Architecture for deploying Vault inside Kubernetes in a highly available manner.
github.com/sethvargo/vault-on-gke
KubernetesEngine
Cloud
KMSCloudStorage
Cloud LoadBalancing
HashiCorp
Vault Server
HashiCorp
Vault Server
HashiCorp
Vault Server
Recommended Architecture Highlights
20
Dedicated GKE Cluster
HashiCorp Vault runs in a dedicated Kubernetes cluster, in a dedicated Google Cloud project to which access is tightly controlled. No other apps run in this cluster.
github.com/sethvargo/vault-on-gke
Google Cloud Storage
Vault stores its encrypted data and configuration in Google Cloud Storage, which enables Vault to run in high availability mode. Google Cloud Spanner is also an option.
Google Cloud KMS
Vault servers are automatically initialized and unsealed by a sidecar container. The initial root token and unseal keys are encrypted with Google Cloud KMS key.
Recommended Architecture Highlights
21
Custom CA for TLS
TLS is managed via a custom certificate authority (CA), All Vault communication should be secured via TLS.
This could be replaced by a trusted CA like Let’s Encrypt.
github.com/sethvargo/vault-on-gke
L4 Load Balancing
GCP L4 regional load balancer delegates TLS to Vault instead of terminating TLS at a L7 load balancer.
This gives Vault full control over TLS and aids in using client certificates as a means for authentication.
Vault as a Service
Vault is available at an IP/DNS address for consumers. K8S is simply an implementation detail.
Specifically, we do not need to leverate K8S-specific features like service discovery.
---apiVersion: apps/v1kind: StatefulSetmetadata: # ...
Deployment Details
StatefulSet
22
The StatefulSet gives us predictable naming, exclusive semantics and a free circuit breaker.
spec: affinity: podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:-labelSelector: matchExpressions:
- key: app operator: In values: [‘vault’]
Deployment Details
Anti-Affinity Rules
23
Require each Vault server runs on a unique host. This provides added availability in case on node goes down.
containers:- name: vault-init image: sethvargo/vault-init:0.1.1
- name: vault image: vault:0.10.3 Args: [‘server’]
Deployment Details
Auto Unsealing
24
A sidecar container automatically initializes and unseals each Vault server.
containers:- # ...
securityContext: capabilities: add: [‘IPC_LOCK’]
Deployment Details
Memory Locking
25
Containers are granted the proper security capabilities to perform memory locking.
readinessProbe: httpGet: path: /v1/sys/health?standbyok=true port: 8200 Scheme: HTTPS initialDelaySeconds: 5 periodSeconds: 5
Deployment Details
Readiness Checks
26
Check the status, allowing standby servers to be “healthy” so they will be added to the load balancer.
---apiVersion: v1kind: Servicespec: type: LoadBalancer loadBalancerIP: 11.22.33.44 Selector: App: vault
Deployment Details
Load Balancer
27
The load balancer binds to the IP address of the external IP address from Google Cloud.
Vault Auth Methods
Vault Tokens
30
User or machines supply information that is validated by Vault. Vault then uses metadata to assign policies to a token.
HashiCorp
Vault Server
User or Machine
Third Party System
Vault Policy3. Map policy to metadata
1.Send information
2. Verify Information
4. Return Vault token
Vault Auth Methods
Auth to Kubernetes
31
Leverage the K8s TokenReviewer API to authenticate a pod to Vault and get a Vault Token.
HashiCorp
Vault Server
Service Account
Vault Policy3. Map policy to metadata
1.Send information
2. Verify Information
4. Return Vault token
TokenReviewer
initContainers: - name: vault-authenticator image: sethvargo/vault-kubernetes-authenticator volumeMounts: - name: vault-token mountPath: /home/vault
Deployment Details
Vault Authenticator
32
An init container pills the pod’s service account from disk, authenticates to Vault, and puts the Vault token inside a shared volume.
Yeah, but... What happens if credentials change? How will I signal my application to re-read the secrets file?
33
X_VAULT_TOKEN=$(</home/vault/.vault-token)
GCP_CREDS=$(curl -k -H "X-Vault-Token: "$X_VAULT_TOKEN"" -X GET "$VAULT_ADDR"/v1/gcp/key/ksdemo-key-roleset |jq -r '.data.private_key_data' | base64 --decode; echo)
echo $GCP_CREDS > /home/demo gcp.json
Deployment Details
Using Credentials
34
Authentication flow with token. Pick up the token, use it to obtain a secret and then apply it for use.