Downtime is not an option - day 2 operations - Jörg Schad

Downtime is not an option How Fast Data and Microservices change the datacenter

@joerg_schad @dcos

Jörg SchadDistributed Systems Engineer

@joerg_schad

In the beginning there was a big

Monolith

Hardware

Operating System

Application

COMPUTERS

noun | ˈmīkrō/ /ˈsərvəs/ :

an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.*

Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable.

*From whatis.com

OVERVIEW

Operating System

ServiceApp ServiceServiceAppApp

MICROSERVICES- Polyglot- Single Responsibility- Smaller Teams- Utilization- Machine

types/groups- Dependency hell

Machine

Infrastructure

Machine Machine

ServiceService ServiceServiceServiceService

Run everything in containers!

CONTAINERS- Rapid deployment- Dependency

vendoring- Container image

repositories- Spreadsheet

scheduling

Machine

Infrastructure

Machine Machine

Container Runtime Container Runtime Container Runtime

CONTAINERSCHEDULING

RESOURCE MANAGEMENT

SERVICE MANAGEMENT

- Load Balancing- Readiness Checking

CONTAINER ORCHESTRATION

CONTAINERSCHEDULING

- Placement- Replication/Scaling- Resurrection- Rescheduling- Rolling Deployment- Upgrades- Downgrades- Collocation

RESOURCE MANAGEMENT

- Memory- CPU- GPU- Volumes- Ports- IPs- Images/Artifacts

SERVICE MANAGEMENT

- Labels- Groups/Namespaces- Dependencies- Load Balancing- Readiness Checking

CONTAINER ORCHESTRATION

Machine Infrastructure

Web Apps & Services

Scheduling

Resource Management

Container Runtime

Machine & OS

Service Management

CONTAINERORCHESTRATION

Machine & OS Machine & OS

Container Runtime Container Runtime

MapReduce is crunching Data

Meanwhile...

But then business demanded

FAST DATAWe need to turn faster!

Fast Data

Batch Event ProcessingMicro-Batch

Days Hours Minutes Seconds Microseconds

Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics

Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations

The SMACK Stack

EVENTSUbiquitous data streams from connected devices

INGEST

Apache Kafka

Apache Spark

ANALYZE

Apache Cassandra

Ingest millions of events per second

Distributed & highly scalable database

Real-time and batch process data

Visualize data and build data driven applications

Mesos/ DC/OS

Sensors

Devices

Clients

Datacenter

NAIVE APPROACH

Typical Datacentersiloed, over-provisioned servers,

low utilization

Industry Average12-15% utilization

microservice

Cassandra

Spark/Hadoop

MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS

Typical Datacentersiloed, over-provisioned servers,

low utilization

Mesos/ DC/OSautomated schedulers, workload multiplexing onto the

same machines

microservice

Cassandra

Spark/Hadoop

• A top-level Apache project• A cluster resource

negotiator• Scalable to 10,000s of

nodes• Fault-tolerant, battle-tested• An SDK for distributed apps• Native Docker support

Apache Mesos

Datacenter Operating System (DC/OS)

Distributed Systems Kernel (Mesos)

DC/OS ENABLES MODERN DISTRIBUTED APPS

Big Data + Analytics EnginesMicroservices (in containers)

Streaming

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern App Components

Any Infrastructure (Physical, Virtual, Cloud)23

THEBASICS

DC/OS is … ● 100% open source (ASL2.0)

+ A big, diverse community● An umbrella for ~30 OSS projects

+ Roadmap and designs+ Docs and tutorials

● Not limited in any way● Familiar, with more features

+ Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages, ...

Container Options

Enhancements to the Mesos Containerizer to allow support launching specific container formats (Docker, AppC, OCI (future), etc)

● Reduces need to maintain and update multiple containerizers

● Support multiple container formats with a single containerizer

Image provisioner component added to the Mesos containerizer - responsible for pulling, caching, and preparing container root filesystems

Launcher Isolators

Universal containerizer

Provisioner

Process management

Container lifecycle hook

Container image support

GEO-ENABLED IoT

DATA FLOW

Keep it running!

Monitoring- Collecting metrics- Routing events- Downstream processing

- Alerting- Dashboards- Storage (long-term retention)

Logging- Scopes- Local vs. Central- Security considerations

DAY 2 OPERATIONS

Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package Management- Networking Policies- Auditing- Backups & Disaster Recovery

Troubleshooting- Debugging

- Services- System- Access?

- Tracing- Chaos Engineering

DAY 2 OPERATIONS

Troubleshooting● Services: typically specific to service, use logging (for

example, dcos task log) and dcos node ssh for

per-node investigations

● dcos task exec

○ Permissions?

● System:

○ Simple diagnostics via dcos node diagnostics

○ Comprehensive dump via clump

THANK YOU!

ANY QUESTIONS?@dcos

users@dcos.io

/groups/8295652

/dcos/dcos/examples/dcos/demos

chat.dcos.io

FailuresFramework

Scheduler

Executor

LEADER STANDBY STANDBY

Executor

Distributed Systems could be so easy...

1. The network is reliable.

2. Latency is zero.

3. Bandwidth is infinite.

4. The network is secure.

5. Topology doesn't change.

6. There is one administrator.

7. Transport cost is zero.

8. The network is homogeneous.

*) https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

Questions?

Code: https://git.io/vXUoy

http://grnh.se/ie76ru

Monitoring

METRICS

Measurements captured to determine health and performance of cluster

- How utilized is the cluster?- Are resources being optimally used?- Is the system performing better or worse over time?- Are there bottlenecks in the system?- What is the response time of applications?

DC/OS METRIC SOURCES

● Mesos metrics ○ Resource, frameworks, masters, agents,

tasks, system, events ● Container Metrics

○ CPU, mem, disk, network● Application Metrics

○ QPS, latency, response time, hits, active users, errors

Container ContainerContainer

App App App

Before upgrading1. Make sure cluster is healthy!2. Perform backup

a. ZKb. Replicated logsc. other state

3. Review release notes4. Generate install bundle

a. Validate versions

UPGRADE PROCEDURE

Framework

Scheduler

Executor

MESOS MASTER METRICS

● Metrics for the master node are available at the following URL:○ http://<mesos-master-ip>/mesos/master/metrics/snapshot ○ The response is a JSON object that contains metrics names and values as

key-value pairs.● Metric Groups:

○ Resources○ Master○ System○ Slaves○ Frameworks○ Tasks○ Messages○ Event Queue○ Registrar

MESOS MASTER BASIC ALERTS

Metric Value Inference

master/uptime_secs is low The master has restarted

master/uptime_secs < 60 for sustained periods of time The cluster has a flapping master node

master/tasks_lost is increasing rapidly Tasks in the cluster are disappearing. Possible causes include hardware failures, bugs in one of the frameworks or bugs in Mesos

master/slaves_active is low Slaves are having trouble connecting to the master

master/cpus_percent > 0.9 for sustained periods of time DCOS Cluster CPU utilization is close to capacity

master/mem_percent > 0.9 for sustained periods of time DCOS Cluster Memory utilization is close to capacity

master/disk_used & master/disk_percent DCOS Disk space consumed by Reservations

master/elected is 0 for sustained periods of time No Master is currently elected

Operations

UPGRADES

1. Master rolling upgradea. Start with standbyb. Install new DC/OS

2. Agent rolling upgrade3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

1. Master rolling upgrade2. Agent rolling upgrade

a. Uninstall DC/OSb. Install new DC/OS

3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

1. Master rolling upgrade2. Agent rolling upgrade3. Framework upgrades

a. Orthogonal to DC/OSb. Ensure changes don’t

affect existing apps

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Failure Handling

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT AGENT

EXECUTOR

Status UpdateStatus Update

EXECUTOR

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

SEGFAULT :(

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

Launch TaskLaunch Task

Launch Task

EXECUTOR

Status UpdateStatus Update

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

Status Update

Failure Handling

MESOS AGENT FAILURE

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

Re-register

Failure Handling

MESOS HOST FAILURE

LOCAL AGENT FAILURE

MASTERMARATHON

EXECUTOR

LOCAL AGENT FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

LOCAL AGENT FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

Status Update

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

Launch TaskLaunch Task

Launch Task

Resource Offer

MESOS TASK FAILURE

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

Status Update

Failure Handling

MESOS MASTER FAILURE

MASTER FAILURE

MASTERMARATHON

EXECUTOR

MASTER FAILURE

MASTERMARATHON

EXECUTOR

MASTER FAILURE

MASTERMARATHON

EXECUTOR

MASTER FAILURE

MASTERMARATHON

EXECUTOR

Leading Master Leading Master

Leading Master

MASTER FAILURE

MASTERMARATHON

EXECUTOR

Reregister

ReregisterReregister

Reregistered

MASTER FAILURE

MASTERMARATHON

EXECUTOR

Reregistered

ReregisteredReregistered

Failure Handling

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

Framework IDLeading Master

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

Reregister

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

Reregistered

SCHEDULER FAILURE

MASTERMARATHON

EXECUTOR

Status Update

Reconcile Tasks

1. Master rolling upgradea. Start with standbyb. Uninstall DC/OSc. Install new DC/OS

2. Agent rolling upgrade3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

1. Master rolling upgrade2. Agent rolling upgrade

a. Uninstall DC/OSb. Install new DC/OS

3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

1. Master rolling upgrade2. Agent rolling upgrade3. Framework upgrades

a. Orthogonal to DC/OSb. Ensure changes don’t

affect existing apps

UPGRADE PROCEDURE

Framework

Scheduler

Executor

FUTURES (TBD)

Leverage maintenance primitives in Mesos to drain host

Upgrade management through DC/OS to perform rolling upgrades

Monitoring- Collecting metrics- Routing events- Downstream processing

- Alerting- Dashboards- Storage (long-term retention)

DAY 2 OPERATIONS

Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package Management- Networking Policies- Auditing- Backups & Disaster Recovery

- Services- System

DAY 2 OPERATIONS

MONITORING

MONITORINGCONCEPT

MONITORINGTOOLINGEXAMPLES

● local scraping:

a. collectd

b. cAdvisor*

● event router:

a. fluentd

b. Flume

c. Kafka*

d. logstash*

e. Riemann

*) available via Mesosphere Universe

● storage:

a. Elasticsearch*

b. Graphite

c. InfluxDB*

d. KairosDB/Cassandra*

e. OpenTSDB/HBase

f. others such a local filesystem, Ceph FS*,

HDFS*, etc.

● dashboard:

b. Grafana*

c. signal fx

● alerting:

a. BigPanda

b. PagerDuty

c. signal fx

d. VictorOps

MONITORINGTOOLINGEXAMPLES(INTEGRATED)

● Amazon CloudWatch ● AppDynamics ● Azure Monitor ● Circonus ● DataDog* ● dcos/metrics● Ganglia ● Google Stackdriver ● Hawkular ● Icinga ● Librato ● Nagios ● New Relic ● OpsGenie ● Pingdom ● Prometheus ● Ruxit Dynatrace* ● Sensu ● Sysdig* ● Zabbix

LOGGING

LOGGINGSCOPES

LOGGINGTOOLINGEXAMPLES(PRIMITIVES) ● DC/OS logging overview

● Docker logging drivers

● systemd's journalctl

LOGGINGTOOLINGEXAMPLES(INTEGRATED)

● Centralized app logging with fluentd

● DC/OS

a. ELK stack log shipping

b. Splunk

● Graylog

● Loggly

● Papertrail

● Sumo Logic

MAINTENANCE

Overview

● How to install a new version of X?● When to scale what (service-level vs. nodes)● Who gets to access/install which services in what way?

Upgrades

Sizing

User and package management

● What services can talk to each other and in which way?● Who accessed what, when and how?● How is the continuous operation of the cluster and the services accomplished?

What happens when cluster (or critical infra components like ZK) go down?

Networking

Auditing

Disaster Recovery

OTHER TROUBLESHOOTING TECHNIQUES

● Tracing

○ Idea: identify latency issues and perform

root-cause analysis in a distributed setup

○ OpenTracing

● Chaos Engineering

○ Idea: proactively break (parts of) the system to

understand how it reacts

○ Chaos Monkey

○ DRAX

ARCHITECTUREMESOS FUNDAMENTALS

● Agents advertise resources to Master● Master offers resources to Framework● Framework rejects/uses resources● Agents report task status to Master

Questions?

Code: https://git.io/vXUoyPsssssssst …… we are hiring!

http://grnh.se/ie76ru

CONTAINER SCHEDULING

RESOURCE MANAGEMENT

SERVICE MANAGEMENT

Service Service Service

Web App Web App Web App

Hardware

Operating System

SERVICE-ORIENTEDARCHITECTURE

- Separation of concerns

- Optimization of bottlenecks

- Smaller teams- API Contracts- Data replication- Complicated

provisioning- Dependency

management

Operating System

Hardware Hardware

Operating System

MICROSERVICES- Polyglot- Single Responsibility- Smaller Teams- Utilization- Machine

types/groups- Dependency hell

Machine

Infrastructure

Machine Machine

THE BIRTH OF MESOS

TWITTER TECH TALKThe grad students working on Mesos

give a tech talk at Twitter.

March 2010

APACHE INCUBATIONMesos enters the Apache Incubator.

Spring 2009

CS262BBen Hindman, Andy Konwinski and

Matei Zaharia create “Nexus” as their CS262B class project.

MESOS PUBLISHEDMesos: A Platform for Fine-Grained

Resource Sharing in the Data Center is published as a technical report.

September 2010

December 2010

April 2016

Monitoring- Collecting metrics

- Routing events- Downstream processing

○ Alerting○ Dashboards○ Storage (long-term retention)

Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package

Management- Networking Policies- Auditing- Backups & Disaster

Recovery

○ Services○ System

Downtime is not an option - day 2 operations - Jörg Schad

Technology

Transcript of Downtime is not an option - day 2 operations - Jörg Schad

Jörg Andres - Rally dos Sertoes

Schad Honors Thesis Final - Stanford University › file › druid:yh056fz1454 › Schad...4) Acknowledgements!!) First)and)foremost,)I)would)like)to)thank)my)thesis)advisor)Teresa)LaFromboise)for)

EAGES Proceedings - G. Jörg

MICHAEL SCHAD, Director of Sales, International · MICHAEL SCHAD, Director of Sales, International LUC PERRAD, Area Sales Manager, DENSO Group Germany Michael joined DENSO Group Germany

Jörg Stelzer

HTTP API In Action by Max Neunhöffer, ArangoDB and Jörg Schad, Mesosphere

Near Zero Downtime

Prof. Dr. Lothar Schad - uni-heidelberg.de · Prof. Dr. Lothar Schad 12/9/2008 | Page 1 Physics of Imaging Systems Basic Principles of Magnetic Resonance Imaging I Prof. Dr. Lothar

Schad v. Arizona, 501 U.S. 624 (1991)

Clinical Downtime Resource Manual - Welcome to LHSC · failures or software failures. Degrees of Downtime The extent of computer downtime may vary. The downtime may impact only certain

Prof. Dr. Lothar Schad - uni-heidelberg.de · 1 Seite 1 RUPRECHT-KARLS-UNIVERSITY HEIDELBERG Computer Assisted Clinical Medicine Prof. Dr. Lothar Schad 12/9/2008 | Page 1 Master‘s

Downtime Tech Mixer

No. 1-09-2079, Langone v. Schad, Diamond and Shedden, P.C.

Understanding Cost Downtime

Xenofon Vasilakospages.cs.aueb.gr/~xvas/pdfs/detailedCV.pdf^downtime _ after handover events. Decreasing mobile users [ perception of service ^Downtime _ (i.e., achieving ^Zero Downtime

Downtime Status (Autosaved)

Peter Schad • Stephen Nortcliﬀ Essentials of Soil …...course, beginning students of soil science. Winfried E. Blum • Peter Schad • Stephen Nortcliﬀ Essentials of Soil Science

Spark Summit EU talk by Jorg Schad

JöRG SCHLAICH

EVA W. SKINNER and EMMA A. SCHAD · EVA W. SKINNER and EMMA A. SCHAD ... of the children in other parts of the world, ... In the homeroom-centered curricu lum, ...