EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

1© Copyright 2016 EMC Corporation. All rights reserved. 1© Copyright 2016 EMC Corporation. All rights reserved.

STATE OF THE CONTAINER ECOSYSTEM WITH PERSISTENT APPLICATIONSSTEVE WONGDEVELOPER ADVOCATE –EMC{code}

2© Copyright 2016 EMC Corporation. All rights reserved.

Old: monolithic, big, slow to change, long lived

APPLICATIONS ARE CHANGINGNew: many instances, rapidly implemented, updated, deployed


SOLUTION: ENTER THE CONTAINER


TODAY’S AGENDATwo rapidly developing areas fields within the container space:1. Persistent

Storage2. Schedulers


WHERE DOES STORAGE FIT INTO THIS?… THE SHELL GAME

Containers and this new “cattle” thing magically made my storage management headaches disappear, right?


IF YOUR CONTAINER ARE CATTLE,YOU HAVE NOTHING TO LOSE …

Go ahead, use local storage. • The test: I don’t need

persistence beyond the life of the container instance or host


I HAVE SERVICES THAT AREN’T CATTLEWHAT IF I RUN THESE THE SAME WAY?

A portion of the container host’s filesystem can be exposed to a container. • Availability Risk

– Migrate container to another host – your storage is gone

– Host goes down – your service goes down• Scale Limitation

– Need more storage than the host has? Sorry…


SOLUTION: MOUNT OF EXTERNAL STORAGE


HOW DO I MOUNT EXTERNAL STORAGE?THE REX-RAY PROJECT

• Open Source - extensible• Supports many container

platforms, including cloud hosted• Supports many storage platforms• Managed and configurable by

container host, not inside-the-container configuration


TODAY’S AGENDATwo rapidly developing areas fields within the container space:1. Persistent Storage2. Schedulers..we’ll come back to storage in a minute


GOT CONTAINERS AT SCALE?YOU NEED A SCHEDULER…


ASSUME 1000’S OF CONTAINERS

• How do you place an assortment of containers of different sizes and shapes onto your servers efficiently?


ASSUME 1000’S OF SERVICES• How do you

find them?• How do they

find each other?


ASSUME 1000’S OF SERVICES


ASSUME 1000’S OF SERVICES

It’s Microservice Time!


WHAT’S A SCHEDULER?• Manages compute nodes

– Takes reports from each backend server, as to whether they are still available, and what their capabilities and capacities are.

• Manages incoming work request queue– Takes user’s, batch, and API job requests– Based on requirements, decides which

backend server(s) can fulfill each request– If multiple backend servers are suitable,

engages optimization algorithms in favor of random placement

on demand

scheduledbatch


MANY SCHEDULERS ARE AVAILABLE• Differences

– Management “philosophy”• “Tell me How” vs “Tell me What”

– aka Imperative vs Declarative, Explicit vs Implicit, declare procedure vs declare outcome– Container technology(s) used– Maturity– Supported scale– Support for external storage volume mounts– Degree of “convergence” with a specific application stack

• Aka degree to which scheduler is “opinionated”


SCHEDULERS COMPARED Swarm Kubernetes Mesos Cloud Foundry

Management style

procedure outcome outcome outcome

Maturity new new years yearsScale 1,000? 1000 10,000+ 5,000Extensibility limited limited Well defined limitedContainer Engine

Docker DockerRoadmap: CoreOS

Linux+DockerRoadmap:CoreOS

Garden

External storage support

DVD RexRay in host RexRay in host RexRay

Application specific framework availability

Roadmap:Aurora for generic

proposed 20+ applications,Marathon and Aurora for generic

Multiple language platforms


KUBERNETES• Started by Google• Primary container integration is with Docker, but

enhancing support for CoreOS in upcoming release• Implicit aka Declarative configuration philosophy

– Declare desired outcome– User defined Key/value labels on objects can be used to

support this


KUBERNETES• pods group containers together’

– pods are the smallest management unit– provides “plumbing” to allow containers to interoperate

(micro-service per container) inside the pods• replication controllers handle container lifecycle• labels provide a way to find and query containers

and other objects• services assemble a set of containers performing a

function


KUBERNETES

• monitors health of services, with auto-healing• Load balancing built in• External volume mounts supported in POC / release

candidate with additional external storage features in active planning stage


SWARM• Started by Docker Inc.• Stays with Docker CLI/API originally developed for

single host, and allows using it across a multiple node cluster

• Explicit aka Imperative configuration philosophy– Declare desired how to achieve outcome

• Compose uses a yaml file to group containers


SWARM• Opportunities for enhancement:

– Container communication across host boundaries– External persistent volume management


MESOS• Mesos is the scheduler that provides maximal

flexibility• It allows you to run sub-schedulers (aka

Frameworks), many of which are available off-the shelf – or you can implement your own

• It can support Docker containers, but also supports non Docker workloads.


MESOS FRAMEWORKS• Marathon

– Supports generic workloads that run in a linux container– Supports Docker containers

• Hadoop• Spark• Kafka• Many more


MESOS STORAGE SUPPORT• External volume mounts supported now• Full global external storage resource management,

using an offer/accept model is in active planning stage now


CLOUD FOUNDRY• Cloud Foundry delivers management at the

application level • Containers are used “under the covers” but you

aren’t exposed to the details and don’t have to manage them


CONTAINER SCHEDULERS AS PAAS STACKS• A variety of PAAS stacks exist that attempt to give

you “everything you need” to create an app on Amazon, or your own hardware, provided you accept the constraints imposed by the platform.

• But marketing people hate to call these constraints. Instead, the PaaS stack is described as opinionated

– Some container schedulers are more opinionated than others. Basically the platform makes technology choices for you, and you don’t get to change these.


CLOUD FOUNDRY• Cloud Foundry is far more opinionated than other

container schedulers – and depending on your objective, and staffing capacity, this could be a good thing


TODAY’S AGENDATwo rapidly developing areas fields within the container space:1. Persistent Storage2. Schedulers..we’ll come back to storage in a minute


A SCHEDULER SHOULD INTERFACE WITH STORAGE AT MULTIPLE LEVELS

Create

Utilize

Retire

Lifecycle management with governance

Mount managementAssignment to applications(tenancy)


WHAT POLLY STORAGE SCHEDULER DOES…• Manages interfaces to storage providers

– Multiple concurrent providers anticipated• Mixed vendors

– Including public cloud– Takes reports from each provider, and what their

capabilities and capacities are– Uses admin defined key/value labels to enable

• Outcome based placement• Concurrent use of multiple schedulers, with quotas• Pool advertisement, with suballocation


STORAGE SCHEDULER = DELIVER SCALE OUTScale out is not just about adding capacity. A Storage Scheduler will let you:• Add, remove, replace storage provider platforms

– in or out of public cloud• Add, remove, replace schedulers• This is about giving you flexibility and

shifting the burden of keeping up to vendors


DEMO

• Demo attach of a persistent volume under Kubernetes

• Shutdown container• Re-attach same volume to a Mesos hosted container,

under Linux containerizer instead of Docker


Data Persistence in the New Container WorldWednesday 3PM

Joshua BernsteinVP of Technology for ETD

Tobi Knaup CEO & Co-Founder of Mesosphere

Guru Session


@EMCcode@cantbewong

emccode.comcommunity.emccode.comcantbewong.com

Come visit us at Booth #1044 or in the vLab

Questions?

EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications

Technology

Transcript of EMC World 2016 - code.13 State of the Container Ecosystem with Persistent Applications