Mesosfiles.meetup.com/12063092/mesos-spark-meetup-04-22-14.pdf · 2014-04-24 · alittlehistory...

Benjamin Hindman – @benh

Mesos for Spark Users mesos.apache.org

@ApacheMesos

agenda ① Mesos

② Spark on Mesos

③ why Mesos? ①  multi-‐tenancy ②  fine-‐grained sharing ③  why not?

④  long-‐lived services and other frameworks

a little history Mesos started as a research project at Berkeley in early 2009 by Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica

our motivation

increase performance and utilization of

clusters

our intuition

①  static partitioning considered harmful

static partitioning considered harmful

datacenter

faster!

higher utilization!

our intuition

②  build new frameworks

"Map/Reduce is a big hammer, but not everything is a nail!"

anatomy of Spark

Driver Program

Spark Context

Worker Node

Executor Cache

Task Task

Worker Node

Executor Cache

Task Task

workers

anatomy of a framework

coordinator

framework ≈

distributed system

workers

coordinator

tasks (and/or executors)

scheduler

execution coordination ==

scheduling

tasks (and/or executors)

Mesos: level of indirection

scheduler

Mesos (slaves)

Mesos: level of indirection

scheduler

Mesos (master)

Mesos: a level of indirection + provide common functionality every new distributed system re-‐implements like failure detection, task distribution, task starting, task monitoring, task killing, task cleanup!

Mesos (slaves)

Mesos: level of abstraction

scheduler

Mesos (master)

Mesos build and run frameworks

using resources

provision and manage machines

build and run frameworks

using resources

deploy and manage applications/services

provision and manage machines

build and run frameworks

using resources

PaaS on Mesos

build and run a PaaS on top of Mesos:

Apache Aurora and Marathon

Mesos on IaaS

use OpenStack or EC2 to run Mesos

Mesos on IaaS++

hardware use OpenStack or EC2 or physical machines

to run Mesos

Mesos: datacenter kernel

hardware

provide common functionality via an API (kernel)

framework

Apache Mesos is a distributed system for running and building other distributed systems

Mesos is a cluster manager

agenda ① Mesos

anatomy of Spark

Driver Program

Spark Context

Worker Node

Executor Cache

Task Task

Worker Node

Executor Cache

Task Task

anatomy of Spark

Driver Program

Spark Context

Worker Node

Executor Cache

Task Task

Worker Node

Executor Cache

Task Task

Cluster Manager

anatomy of Spark

Driver Program

Spark Context

Worker Node

Executor Cache

Task Task

Worker Node

Executor Cache

Task Task

Mesos is a distributed system with a master/slave architecture

masters

slaves

Mesos challenged the status quo of cluster managers

cluster manager status quo

cluster manager

application

specification

the specification includes as much information as possible to assist the cluster manager in scheduling and execution

cluster manager

application wait for task

to be executed

cluster manager

application

result

problems with specifications

① hard to specify certain desires or constraints

② hard to update specifications dynamically as tasks executed and finished/failed

an alternative model

masters

framework

request 3 CPUs 2 GB RAM

a request is purposely simplified subset of a specification, mainly including the required resources

question: what should you do if you can’t satisfy a request?

①  wait until you can …

②  offer best you can immediately

Mesos model

masters

framework

offer hostname 4 CPUs 4 GB RAM

Mesos model

masters

framework

an analogue: non-‐blocking sockets

kernel

application

write(s, buffer, size);!

an analogue: non-‐blocking sockets

kernel

application

42 of 100 bytes written!!

offers represent the current snapshot of available resources a framework can use

(requests are complimentary, but not necessary; see Google’s Omega)

masters

Driver Program

Spark Context

Spark on Mesos

masters

Driver Program

Spark Context

Spark on Mesos

masters

Driver Program

Spark Context

Spark on Mesos

masters

Spark uses the offers to perform it’s own scheduling

Driver Program

Spark Context

Spark on Mesos

masters

task 3 CPUs 2 GB RAM

Driver Program

Spark Context

Spark on Mesos

masters

Driver Program

Spark Context

“two-‐level scheduling”

Spark on Mesos

“two-‐level scheduling” Mesos: controls resource allocations to Spark

Spark: makes decisions about what tasks to run given available resources

masters

framework

frameworks launch fine-‐grained tasks for execution

execution

masters

framework

task + executor 3 CPUs 3 GB RAM

frameworks launch fine-‐grained tasks for execution

if necessary, a framework can provide an executor to handle the execution of a task

execution

a task with a command

mesos-slave!

task! task!

a task with an executor

mesos-slave!

executor!

mesos-slave!

executor!

mesos-slave!

executor!

task! task!

mesos-slave!

executor!

mesos-slave!

executor!

task! task!

Spark execution

masters

task + executor 1 CPUs 3 GB RAM

Driver Program

Spark Context

Spark execution

mesos-slave!

Spark execution

mesos-slave!

Executor Cache

Spark execution

mesos-slave!

Executor Cache

Spark execution

mesos-slave!

Executor Cache

Task Task

Spark execution

mesos-slave!

Executor Cache

Task Task

Executor Cache

resource isolation

Mesos has containerization support on Linux (built-‐in usage of cgroups and namespaces)

isolator modules: CPU (upper and lower bounds) memory network I/O (in development) filesystem (using LVM, planned)

resource isolation

mesos-slave!

executor!

resource isolation

mesos-slave!

executor!

containers

resource isolation

mesos-slave!

Executor Cache

Task Task

Executor Cache

agenda ① Mesos

multi-‐tenancy

slave(s)

mesos-slave!mesos-slave!

multi-‐tenancy (only Spark)

slave(s)

(can approximate w/ standalone mode by setting max # cores per application, otherwise get FIFO execution)

multi-‐tenancy (only Spark)

slave(s)

(run the tried and true and test out the new at the same time!)

0.9.0 0.8.1 0.8.1 0.9.0

agenda ① Mesos

fine-‐grained sharing

mesos-slave!

Executor Cache

Task Task

Executor Cache

Task Task

Spark executors only consume memory, can share CPU between

mesos-slave!

Executor Cache

Task Task

Executor Cache

mesos-slave!

Executor Cache

Task Task

mesos-slave!

Executor Cache

Task Task

mesos-slave!

Executor Cache Executor Cache

Task Task Task Task

mesos-slave!

Task Task Task

mesos-slave!

Task Task Task Task

mesos-slave!

Task Task Task

mesos-slave!

Task Task Task Task

mesos-slave!

Task Task Task

mesos-slave!

Task Task

agenda ① Mesos

why not?

more moving pieces means more things to learn and more things that can fail …

masters

master failover

framework

after a new master is elected all frameworks and slaves connect to the new master

all tasks keep running across master failover!

slave failover

mesos-slave!

task! task!

slave failover

mesos-slave!

task!task!

slave failover

task!task!

slave failover

mesos-slave!

task!task!

slave failover

mesos-slave!

task!task!

slave failover @twitter

mesos-slave!

(large in-‐memory services, expensive to restart)

masters

framework failover

framework

framework re-‐registers with master and resumes operation

all tasks keep running across framework failover!

framework

agenda ① Mesos

Apache Aurora (incubating)

masters

Aurora is a Mesos framework that makes it easy to launch services written in Ruby, Java, Scala, Python, Go, etc!

Marathon (from Mesosphere)

masters

Marathon is a Mesos framework that makes it easy to launch services written in Ruby, Java, Scala, Python, Go, etc!

Marathon

Jenkins on Mesos

(http://bit.ly/1frLrLf)

(http://bit.ly/1nHwM3r)

Apache Mesos and Jenkins -‐ elastic build slaves

Elastic Mesos: elastic.mesosphere.io

Packages:

Thank You!

mesos.apache.org

mesos.apache.org/blog

@ApacheMesos

Mesosfiles.meetup.com/12063092/mesos-spark-meetup-04-22-14.pdf · 2014-04-24 · alittlehistory...

Documents

Transcript of Mesosfiles.meetup.com/12063092/mesos-spark-meetup-04-22-14.pdf · 2014-04-24 · alittlehistory...

Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center · 2017-09-12 · Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center Benjamin Hindman, Andy

Mesos: A Platform for Fine-Grained Resource Sharing in the ... · Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center Benjamin Hindman, Andy Konwinski, Matei Zaharia,

THE DATACENTER NEEDS AN OPERATING SYSTEM MATEI ZAHARIA, BENJAMIN HINDMAN, ANDY KONWINSKI, ALI GHODSI, ANTHONY JOSEPH, RANDY KATZ, SCOTT SHENKER, ION STOICA.

Mesos introduction

Q -SERVICE-AWARE S HETEROGENEOUS D PARAGONdelimitrou/papers/2014.toppicks.paragon.pdf8. B. Hindman et al., “Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center,”

Apache Mesos

Leslie Hindman Auctioneers Sale_468

Benjamin Hindman – @benh Apache Mesos Design Decisions mesos.apache.org @ApacheMesos.

Session 5 ic2011 hindman

Dominant Resource Fairness: Fair Allocation of Multiple Resource Types Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, Ion.

Hadoop on-mesos

capadriaics.files.wordpress.com...PEDIATRIA 2016: preventives de la infància i l'adolescència. Calendari de Revisions Pediàtriques EDAT Oa 30 DIES 2 MESOS 4 MESOS 6 MESOS MESOS

ApacheMesos - UC Berkeley AMP Camp | The UC Berkeley Big ...ampcamp.berkeley.edu/wp-content/uploads/...camp-2012-mesos-at-t… · origins Berkeley(research(project(including(Benjamin(Hindman,Andy(Konwinski,Matei(Zaharia,Ali

Mesos and YARN

A Whirlwind Tour of Apache Mesos - YOW! Conferences · Mesos Analogy to an Operating System Linux Mesos. Mesos vs Virtualization Virtualization Mesos. Mesos Architecture •ZooKeeper

Terraform + Mesos

Docker on Mesos

Mesos Go Stateful - events.static.linuxfound.org · Analysis of Different Stateful Workload MySql Kafka ETCD PostgreSql Redis ... Mesos Go Stateful Mesos Go Framework Executor Executor

Mesos A Platform for Fine-Grained Resource Sharing in the Data Center Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony Joseph, Randy.

Mesos ♥ Docker