PushingKubernetes Forward

Brandon Philips@brandonphilips | brandon@coreos.com | coreos.com

CoreOS, Inc (2013 - today)Mission: "Secure the Internet"

Started at the OS level: CoreOS Linux● Modern, minimal operating system● Self-updating (read-only) image● Updates must be automatic and seamless

app1app2app3

server1

app4app5

server2

app6app7

server3

app1app2app3

server1

app4app5

server2

app6app7

server3

updating...

app1app2app3

server1

app4app5

server2

app6app7

server3

needs reboot

app1app2app3

server1

app4app5

server2

app6app7

server3

rebooting...

Without orchestration

app1app2app3

server1

app4app5

server2

app6app7

server3

rebooting...

Without orchestration

app1app2app3

server1

app4app5

server2

app6app7

server3

needs reboot

With orchestrationmagical

orchestrator

server1

app4app5app1app3

server2

app6app7app2

server3

needs reboot

With orchestrationmagical

orchestrator

server1

app4app5app1app3

server2

app6app7app2

server3

rebooting...

With orchestrationmagical

orchestrator

server1

app4app5app1app3

server2

app6app7app2

server3

updated!

With orchestrationmagical

orchestrator

app2app3

server1

app4app5app1

server2

app6app7

server3

updated!

With orchestrationmagical

orchestrator

app2app3

server1

app4app5app1

server2

app6app7

server3

With orchestration

90+ Projects on GitHub, 1,000+ Contributors

OPEN SOURCE

CoreOS.com - @coreoslinux - github/coreos

Secure solutions, support plans, training + more

ENTERPRISE

sales@coreos.com - tectonic.com - quay.io

Product Management via Keynote

Users running Kubernetes infrastructure

Community building Kubernetes

Businesses building products on Kubernetes

Where We Are Pushing Kubernetes

Simpler to deploy and configure clusters

Increasing scale of clusters throughout stack

Security based on good practices

rkt engine powering Kubernetes nodes

Standards to ensure portability

Simpler Deploymentself-hosted k8s

workerkubelet

workerkubelet

workerkubelet

scheduler& API

workerkubelet

workerkubelet

workerkubelet

workerkubelet

API Server

scheduler controllermanager

And a few more pieces in containers

DNS addon replica set

Heapster and InfluxDB

Networking daemon set

Identity and authz services

How do we install it all?

Manually place configuration

Cloud-config and bash

Config management

How do we install it all?

Manually place configuration

Cloud-config and bash

Config management

How do we upgrade it all?

$ monokube --nodes=172.17.8.101,172.17.8.102,...

monokube - a prototype

ssh reverse tunnel

$ monokube --nodes=172.17.8.101,172.17.8.102,...

monokube - a prototype

deploy API server

$ monokube --nodes=172.17.8.101,172.17.8.102,...

monokube - a prototype

re-configure API cfg

Self-hosting Kubernetes Pivot

kubectl

Self-hosting Kubernetes Pivot

That seems hard, what do we get?

Bootstrap requirements down to working SSH

Rolling updates for Kubernetes itself!

Kubelet version controlled by API

Help Wanted! Goal: working in v1.3

Simpler Deploymentjoin sig-high-availability

Increasing Scalescheduler improvements

https://coreos.com/blog/improving-kubernetes-scheduler-performance.html

10x Improvement in scheduler throughput

Ongoing work to track upstream

performance

Let's make similarly large gains in v1.3

Help wanted: Kubemark dashboard!

Increasing Scaleetcd v3 in k8s

etcd v3.0 - "Scaling etcd to thousands of nodes"

● Efficient transport via gRPC and HTTP/2● New powerful API based on k8s use-case● Disk-backed and memory efficient storage● Incremental snapshot for consistent performance● Fix re-list issues with longer and memory-efficient

key history

v3 API - Transactions

● compare and swap○ compare: foo=bar○ success: foo=bar2

● multiple object transaction○ compare: cond1=true && cond2=true○ success: pass=true○ failure: pass=false

v3 API - Watches

● support multiple keys and prefixes per stream

○ watchKey(foo)

○ watchPrefix(coreos)

● support watch from historical point

○ watchKey(foo, index_of_an_hour_ago)

○ user-driven history compaction

v3 API - Lease

l := lease.Create(10*second)

kv.Put("foo", "bar", l.ID)

// key will be removed without keeping // alive the leasego KeepAlive(l.id)

Help Wanted: mirror maker

Label queries are the new DNS

Need API mirrors to give queries 100% uptime

Help wanted, no work started.

When is the release?

When is it in k8s?

● etcd v3 k8s issue #22448○ Refactoring the storage interface○ Proof of concept working

Increasing Scalejoin sig-scalability

Security Through IdentityOIDC in Kubernetes

Dex - OIDC Provider

Open source standards based identity-providerSQL, LDAP, and other identity backend connectorsApplicable outside of Kubernetes but that is our use case

OIDC

Relying Party

OIDC

End User

OIDC

Identity Provider

OIDC

Identity Provider

OIDC

Identity Provider

OIDC

Identity Provider

OIDC

0. Relying party periodically syncs public key from IdP

1. User request protected page

OIDC

2. User redirected to auth page

OIDC

3. User authenticates (cookie/pw)

OIDC

4. User given authz grant

OIDC

5. User presents grant to client

OIDC

6. Relying party exchanges authz code for ID token

OIDC

7. Client gets ID token and validate claims

OIDC

JWTJSON Web Token

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.

eyJzdWIiOiIyNDgyODk3NjEwMDEiLCJuYW1lI

joiSmFuZSBEb2UiL...

mphbmVkb2VAZXhhbXBsZS5jb20iLCJwaWN

0dXJlIjoiaHR0cDovL2V4YW1wbGUuY29tL2ph

bmVkb2UvbWUuanBnIn0.

TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeF

ONFh7HgQ

{

"alg": "HS256",

"typ": "JWT"

}

{

"sub": "248289761001",

"name": "Ada Richmond",

"preferred_username": "ada",

"email": "ada.richmond@example.com",

"groups": ["read-prod", "admin-stage"]

}

Groups and Kubernetes

API server extracts user, email, groups, from OIDC token

Now what?

Webhook Authorizer "kind": "SubjectAccessReview", "spec": { "resourceAttributes": { "namespace": "default", "verb": "GET", "group": "group3", "resource": "pods" }, "user": "ada", "group": ["read-prod", "admin-stage" ] } authorizer service

OK?

Security Through IdentityOIDC in Kubernetes

rkt Powered Kubernetesmid-flight engine swap

a modern, secure container runtimea simple, composable tool

focused on kubernetes

no central daemonno (mandatory) API

apps run directly under spawning process

rkt - simple CLI tool

bash/systemd/kubelet

rkt run ...

application(s)

modular architecturetake advantage of different technologies provide a consistent experience to users

rkt internals

Nearly complete!80% of end-to-end tests passingcAdvisor integration in progress

rktnetes today

LIVE DEMO

rktnetes today

Goal: 100% end-to-end tests working

User may switch to rktnetes with zero suprises

rktnetes today

rkt Powered Kubernetesjoin sig-node

Security TPM Log

● TPM, Trusted Platform Module○ physical chip on the motherboard○ cryptographic keys + processor

● Used to "measure" system state● Historically just use to verify bootloader/OS (on

proprietary systems)

rkt TPM measurement

● CoreOS added support to GNU Grub● rkt can now record information about running

pods in the TPM● attestable record of what images and pods are

running on a system

rkt TPM measurement

rkt TPM measurement

https://coreos.com/blog/coreos-trusted-computing.html

Tectonic Trusted Computing

TPM Attestation in k8s

1. Generated timestamp2. Ask TPM for sig of time

+ log value3. Submit to API server in

nodeStatus

TPM Attestation in k8s

Goal: Merge nodeStatus payload upstream in k8s v1.3

rkt TPM measurement

For more TPM and rkt, see Matthew Garrett's talk:

"Integrated trusted computing in Kubernetes" 11:30am today

Security TLS Bootstrap

TLS Bootstrap of Nodes (#20439)

1. Generate CSR2. Submit CSR to API

server3. Poll for approved CSR

TLS Bootstrap of Nodes (#20439)

Goal: Merge proposal and working code into v1.3

Industry Movement

● Coordinate promotion of Cloud Native architectures● A home for Cloud Native OSS projects like Kubernetes

○ Technical board to evaluate additional projects● Provides shared resources to projects like video

conferencing, test servers, etc

● Creating technical standards for containers● Started with runC and a runtime specification● Large mandate to standardize an image format

○ In-progress

Multiple Image Formats in v1.3 API

● Today Kubernetes only supports the Docker Image Format and naming

● Use cases for executing other formats○ OCI Image Format○ tar archive chroots○ jar?○ static binary?

● Support signing and content verification

Help Push Kubernetes Forward

Simpler to deploy and configure clusters

Increasing scale of clusters throughout stack

Security based on good practices

rkt engine powering Kubernetes nodes

Standards to ensure portability

coreos.com/fest - @coreosfestMay 9 & 10, 2016 - Berlin, Germany

Thank you!

Brandon Philips@brandonphilips | brandon@coreos.com | coreos.com

We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers