Matt TavisSr Staff Solutions Architect, GoogleZach CasperHybrid Solution Lead, Kubernetes, Google

Google has been developing and using containers to manage our applications for over 12 years.

Images by Connie Zhou

Everything at Google runs in containers:• Gmail, Web Search, Maps, ...• MapReduce, batch, ...• GFS, Colossus, ...• Even Google’s Cloud Platform:

our VMs run in containers!

Why Kubernetes? Workload Portability

Goal: Avoid vendor lock-in

Runs in many environments, including “bare metal” and “your laptop”

The API and the implementation are 100% open

The whole system is modular and replaceable

Workload portability

Goal: Avoid coupling

Don’t force apps to know about concepts that are Kubernetes-specific

Examples of this:

● Namespaces● Services / DNS● Downward API● Secrets / ConfigMaps

Workload portability

Why GKE?

Fully managed Kubernetes for deploying applications

Stateful app support Docker image support

Security & compliance

OS built for containers

Private container registry

Fast, consistent builds

Open source portability

Resource limits Identity & access management

AutoscaleHybrid networking Auto-upgradeFully managed

Integrated logging & monitoring

Auto-repair

Enterprise Readiness Developer Experience

Google is your SRE

Container EngineKubernetes Master

API Server

Controller Manager

Scheduler

etcd

Google manages your control plane

● Backups● Monitoring● Restarts● Resizing for larger clusters

Container EngineKubernetes Master

API Server

Controller Manager

Scheduler

etcd

...and components on your nodes

Container EngineKubernetes Nodes

Logging

Monitoring

Ingress backend

Runtimes

Your App

Node upgrade:

● Update Kubernetes version

● Update node OS

Node repair:

● Automatically repair broken nodes

Let Google manage your nodes!

Container EngineKubernetes Nodes

Logging

Monitoring

Ingress backend

Runtimes

Your App

Autoscaling

Automatically add (or remove) pods as needed

‒ Based on CPU utilization (for now)‒ Custom metrics from Stackdriver

● Efficiency now, capacity when you need it● Operates within user-defined min/max bounds● Set it and forget it

Horizontal Pod Autoscaling

...

Stats

Add VMs when needed

‒ Based on unschedulable pods‒ New VMs self-register with API server

Remove VMs when not needed

● e.g. CPU usage too low...

Cluster autoscaling

IAM and RBAC in GKE

Full audit trail

All GKE requests logged in the operations list

All Kubernetes API requests logged to Stackdriver

GCP Native Dashboard

CI/CD Tools

Google Cloud Container Registry

Container Registry is a fast, private Docker image storage on Google Cloud Platform

● Inherits benefit of availability, durability of GCP hosting● Charged based on overall storage of images, not on # of

images or # of repos● Easily hooks into other GCP services

○ Deploy containers to GKE, GAE, GCE○ Lock down permissioning on what systems or users

can access images using IAM● ALPHA feature of being able to scan containers for

vulnerabilities

Google Cloud Container Builder

Container Builder is a service that executes your builds on Google Cloud Platform

● Can import source code from GCS, CSR, Github, or Bitbucket

● Produces artifacts like Docker containers or Java archives● Executes build as a series of build steps, there are default

steps or you can write your own● Builds are initiated by triggers and serverless so no need to

manage build VM fleets

Google Cloud Container Builder (example)

Directory Structure:

(root directory name) |-- cloudbuild.yaml -- Dockerfile -- quickstart.sh

Files:

quickstart.sh

Dockerfile

cloudbuild.yaml

Google Cloud Container BuilderBuild History Build Details

Spinnaker

Spinnaker is an open-source, multi-cloud, continuous delivery platform for releasing software changes with high velocity and confidence.

Spinnaker provides cluster management as well as deployment management within the cloud.

Spinnaker: Deployment Management

Allows users to construct and manage continuous delivery workflows using pipelines and stages.

Pipelines consist of a sequence of actions, known as stages.

Observability Tools

Stackdriver Overview

Monitoring, logging, and diagnostics for applications on Google Cloud

Platform and Amazon Web Services

Monitoring

Monitoring for applications running on Google Cloud Platform

Logging

Logging for applications running on Google Cloud Platform and

Amazon Web Services

Error Reporting

Identify and understand your application errors

Trace

Find performance bottlenecks in production

Debugger

Investigate your code’s behavior in production

Beyond GKE

Open Hybrid Cloud

Kubernetes

Open platform for running containers, portable apps that run across environments

Run Open Services

Simple, elegant way to deliver and consume services across environments

Managed by policy

Connect, manage, and secure services across environments

Istio

Open Service Broker API

• Provides a standard way to instantiate and consume any service

• Producers (e.g., GCP) create brokers to provide services

• Consumers can discover and access via a catalog that subscribes to these brokers

CloudSQL

BigQuery Email Storage

PubSub Storage SQL Chat

What is open service broker?Service provisioning and discovery

1. Create a service broker

2. Browse service catalog

3. Create service instance; choose your plan size

4. Bind the service to your app

5. Unbind the service when you’re done

6. Delete the service BigQuery

CloudPub/Sub

CloudBigtable

CloudSQL

CloudSpanner

CloudStorage

Kubernetes, Service Catalog and OSB integration

Securing service traffic

Uniform observability

Operational agility

Service Discovery Load Balancing A/B testing

Failure Recovery End to end authentication Canary releases

Metrics / Monitoring Rate Limiting Access Control

Istio Service MeshService management

Istio Architecture Overview

● Pilot: Control plane to configure and push service communication policies.

● Envoy: Network proxy to intercept communication and apply policies.

● Mixer: Policy enforcement with a flexible plugin model for providers for a policy.

● Istio Auth: Service-to-service auth[n,z] using mutual TLS, with built-in identity and credential management.

Control Plane API

Mixer

Service A Service B

proxy proxy

HTTP/1.1, HTTP/2, gRPC or TCP -- with or without

mTLS

Pilot Istio Auth

Config data to Envoys

TLS certs to Envoys

Policy checks, telemetry

Istio Multi Cluster support

svcA

Envoy

Pod

Service A

svcB

Envoy

Service B

Pilot Mixer

Discovery & Config data to Envoys

Citadel

svcC

Envoy

Service C

Primary/Local Kubernetes deployment Secondary/Remote Kubernetes deployment (s)

ApigeeService provisioning and discovery

Management services

Analytics

Dev management

Security analytics

Monolithic systemson premise

Centralized governance of all API services

Kubernetes IntegrationAll Apigee services are Kubernetes services

• Integrating legacy applications• Recompose monolithic applications as services• Build a service layer in front of existing systems to increase IT velocity• Import legacy systems into modern, container-based architectures as services

Users or other Cloud Services

Infrastructure

Istoi Service Mesh

Kubernetes

ManagedCloud Services

Legacy Application Services

Cloud Infrastructure On-Prem Infrastructure

Cloud NativeServices

Thank you.