Mesosphere // Data Driven NYC // September 2014

29
Introduction to the Mesosphere Stack Tobi Knaup [email protected] @superguenter

description

Mesosphere Co-Founder and CTO Tobi Knaup presented at September 2014's edition of Data Driven NYC. Mesosphere turns data centers into one big computer.

Transcript of Mesosphere // Data Driven NYC // September 2014

Page 1: Mesosphere // Data Driven NYC // September 2014

Brand GuidelinesVersion 110.21.2013

Introduction to the Mesosphere Stack

Tobi Knaup [email protected] @superguenter

Page 2: Mesosphere // Data Driven NYC // September 2014

Who am I?

Co-Founder & CTO at Mesosphere

Engineer 4 at Airbnb, tech lead, built infrastructure/team

Grad school work on sentiment detection with big(ger) data

Page 3: Mesosphere // Data Driven NYC // September 2014

Imagine if…

Page 4: Mesosphere // Data Driven NYC // September 2014

All your servers

Page 5: Mesosphere // Data Driven NYC // September 2014

were pooled together

Page 6: Mesosphere // Data Driven NYC // September 2014

So they behave like one big computer

Page 7: Mesosphere // Data Driven NYC // September 2014

as easy as

Page 8: Mesosphere // Data Driven NYC // September 2014

…and building new

datacenter apps is as easy as building an

app for one machine

PowerBook G4

Page 9: Mesosphere // Data Driven NYC // September 2014

Fantasy?

Page 10: Mesosphere // Data Driven NYC // September 2014

Datacenter Operating System

Brand GuidelinesVersion 110.21.2013

Page 11: Mesosphere // Data Driven NYC // September 2014

Why should I care?

Big data analysts today use a multitude of tools

Mesos makes it easy to run them all

!

Building big data tools means building distributed systems

Mesos is an SDK for building distributed systems

!

Tuning clusters for efficiency is a lot of work

Mesos automatically bin-packs apps to increase utilization

Page 12: Mesosphere // Data Driven NYC // September 2014

Applications in the Cloud Era

Cloud Era:Big apps, small servers

Client-Server Era:Small apps, big servers

Server

Virtualization

App App App AppApp

Aggregation

Serv Serv Serv Serv

Page 13: Mesosphere // Data Driven NYC // September 2014

Enterprise Technologies pioneered by Google

Big Data MapReduce (2004) ➞ Hadoop (2005) ➞ Cloudera (2008)

NoSQL Bigtable (2006) ➞ Cassandra (2008) ➞ DataStax (2010)

Datacenter Operating System The Datacenter as a Computer (2009) ➞ Mesos (2009) ➞ Mesosphere (2013)

Page 14: Mesosphere // Data Driven NYC // September 2014

The UNIX Operating System Stack

SSHd

Linux, BSD

MySQLApache

Kernel

ApplicationsMemcached

Init, Upstart, Systemd Init System

Page 15: Mesosphere // Data Driven NYC // September 2014

The Mesosphere Operating System Stack

Memcached

Mesos

RedisRails

Kernel

ApplicationsElasticsearch

Marathon Init System

Page 16: Mesosphere // Data Driven NYC // September 2014

The Mesosphere Operating System Stack

Kernel Mesos

Spark, Storm, Hadoop, ElasticSearch, MPI

Batch REST API“Chronos” (~cron)

Services REST API“Marathon” (~init)

Rails, Kafka, Play!(any that runs on Linux)

Mesos SDKJava, Python, C++, Go

Recurring JobsETL, Hadoop, Backups

API

Apps

Hardware Server Server Server Server Server Server Server

Native Long Running/Linux Batch/Linux

Page 17: Mesosphere // Data Driven NYC // September 2014

Mesos as a Distributed Operating System Kernel

Cluster level resource scheduler

Launch tasks across the cluster (like threads)

Failure detection

Communication between tasks (like IPC)

Distributed state

APIs for building “native” applications (frameworks)

Custom frameworks are elastic from the start

Easy failover and HA

Page 18: Mesosphere // Data Driven NYC // September 2014

Core Technology

Scales to 10,000s of nodes, production grade

Mesos is Top-level Apache project

Mesosphere, Twitter and Airbnb are major users / contributors

Built-in containerization, incl. Docker

Page 19: Mesosphere // Data Driven NYC // September 2014

Mesos - Frameworks

Page 20: Mesosphere // Data Driven NYC // September 2014

Chronos

Page 21: Mesosphere // Data Driven NYC // September 2014

Data Infrastructure @ Airbnb pre Chronos

DBDB

DBProd DB

Analytics DBAnalytics

DBRedShift

Elastic MapReduce

RedShiftRedShift

CRON

S3 PigHive Cascading Cascalog

Page 22: Mesosphere // Data Driven NYC // September 2014

create_omg_table-fact_ad_stats create_omg_table-dim_ad_groups

create_omg_table-ad_stats

create_tmp_users_visits_history_table

create_omg_table-dim_dates

create_airbed_dump_table_reservation2s

create_omg_table-dim_keyword_templates

create_airbed_dump_table_users

bookings_history

hostings_summary_2_quality_score

daily_omg_jobs-ad_stats_ad_weekly

daily_gibson-import_omg_tables

daily_omg_jobs-ad_stats_kw_summary

daily_gibson-default_data_cooked_table_import

daily_omg_jobs-ad_stats_combined

create_omg_table-ltv_user_revenue_yr1_fc

hostings_summary

hostings_impressions_normalize_prepare

high_level_report_hs_history

hostings_summary_1_pre

hostings_impressions_normalize

update_users_visits_history_table

daily_omg_jobs-omg_adwords_report_next30

create_users_visits_history_table

users_summary

daily_omg_jobs-ad_stats_af_market

Page 23: Mesosphere // Data Driven NYC // September 2014

Mesos - Frameworks

Page 24: Mesosphere // Data Driven NYC // September 2014

Problems

10,000 lines of CRON / BASH

“sleep 300”

AWS: unreliable network / timeouts

Cron - not HA, single node

DB Import / Export means heavy I/O (single node)

Debugging hard, 12 hrs / week!

Page 25: Mesosphere // Data Driven NYC // September 2014

Requirements for building Chronos

Existing tools too heavyweight

Express & visualize dependencies

Retries, fault tolerance, HA

Distributed, elastic

Raw BASH (for doing Hadoop, Rake, etc.)

Debugging hard, 12 hrs / week!

REST endpoint (easy to script against)

Page 26: Mesosphere // Data Driven NYC // September 2014

Building Chronos

Mesos framework

CRON for the data center

3K lines of Scala

ISO8601 interval notation (R10/2013-02-28T14:00:00Z/P1D)

Elastic, HA

No network programming

Page 27: Mesosphere // Data Driven NYC // September 2014

Architecture with Chronos

Mesos SlaveMesos Slave

ChronosChronos

Mesos SlaveMesos Slave

job

Mesos Master

DBDB

DBProd DB

Analytics DBAnalytics

DBRedShift

Elastic MapReduce

RedShiftRedShift

S3

PigHive Cascading Cascalog

Mesos Master

jobnotification

Page 28: Mesosphere // Data Driven NYC // September 2014
Page 29: Mesosphere // Data Driven NYC // September 2014

Deployments