Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert
-
Upload
mesosphere-inc -
Category
Software
-
view
3.487 -
download
2
description
Transcript of Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert
![Page 1: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/1.jpg)
Brand GuidelinesVersion 110.21.2013
Software for the hyperscale datacenter
Florian Leibert, CEO & Founder [email protected]
![Page 2: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/2.jpg)
Imagine if…
![Page 3: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/3.jpg)
All your servers in your datacenter
and cloud
![Page 4: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/4.jpg)
were pooled together
![Page 5: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/5.jpg)
So they behave like one big computer
![Page 6: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/6.jpg)
as easy as
![Page 7: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/7.jpg)
…and building
new datacenter apps is as
easy as building an app for one
machine PowerBook G4
![Page 8: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/8.jpg)
Fantasy?
![Page 9: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/9.jpg)
![Page 10: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/10.jpg)
Brand GuidelinesVersion 110.21.2013
![Page 11: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/11.jpg)
Mesos Users Today
Sigmoid Analytics
![Page 12: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/12.jpg)
![Page 13: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/13.jpg)
Today’s Legacy Datacenter
![Page 14: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/14.jpg)
Provision VMs in the cloud or on physical servers
Today’s Legacy Datacenter
![Page 15: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/15.jpg)
Install Hadoop on a static set of machines
Installing an Application with Static Partitioning
![Page 16: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/16.jpg)
Install Web Server on a static set of machines
Installing an Application with Static Partitioning
![Page 17: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/17.jpg)
Scale up Hadoop manually
Resizing an Application with Static Partitioning
![Page 18: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/18.jpg)
What if your Laptop was operated like your Data
![Page 19: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/19.jpg)
Issues with Statically Partitioned Data Centers
Complex Machine sprawl, manual resize/scale LimitedNo software failure handling, “black box”
Inefficient Static partitioning, overhead
Not Developer-FriendlyLong time to roll out software, development starts at the machine level
![Page 20: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/20.jpg)
Aggregation
Mesosphere aggregates resources, makes a data center look like one big computer
Mesosphere runs on top of a VM or on bare metal
![Page 21: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/21.jpg)
Applications in the Cloud Era
Cloud Era: Big apps, small servers
Client-Server Era: Small apps, big servers
Server
Virtualization
App App App AppApp
Aggregation
Serv Serv Serv Serv
![Page 22: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/22.jpg)
From Static Partitioning to Elastic Sharing
Static Partitioning
Elastic Sharing
WEB HADOOPCACHE
WASTED
FREEFREEHADOOP
WEB
CACHE
WASTED WASTED100% —
100% —
![Page 23: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/23.jpg)
Applications are Changing
![Page 24: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/24.jpg)
Deployments
![Page 25: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/25.jpg)
Mesos Facts
Scales to 10,000s of nodes Top-level Apache project Twitter and Airbnb are major users and contributors APIs for C++, Python, JVM, Go Pluggable CPU, memory, IO isolation Packages and commercial support through Mesosphere Highly available, scalable, elastic
![Page 26: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/26.jpg)
Apache Mesos Features
Multi-tenancy
Improved resource utilization
Resource Isolation
Fault-tolerance, HA
Scalability + Elasticity
26
![Page 27: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/27.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
![Page 28: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/28.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
28
![Page 29: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/29.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
29
![Page 30: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/30.jpg)
Mesos Slave
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ./ruby XYZ
Mesos Slave
Docker Executor Docker Executor
java -jar XYZ.jar ./xyz
Mesos Master Mesos Master Mesos Master
Hadoop scheduler
Marathon scheduler
Zookeeperquorum
30
![Page 31: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/31.jpg)
Resource Offers and Launching a Task
1. resourceOffers()2. launchTasks()
3. launchTask() 4. statusUpdate()
5. statusUpdate()
Mesos Slave
Executor
Mesos Master
Scheduler
Task Task
![Page 32: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/32.jpg)
Response times and overhead are significantly Ti
me
to p
rovi
sion
(sec
onds
)
1
100
10000
Bare metal VM Container
Inspired by Tomas Barton’s Mesos talk at InstallFest in Prague
![Page 33: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/33.jpg)
Mesos provides fine-grained resource isolation
Compute Node
Mesos Slave Process
Hadoop task-tracker Mesos Executor
Task #1 Task #2 ruby XYZ
Container(Cgroups)
Executor
![Page 34: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/34.jpg)
Mesos provides fine-grained resource isolation
Compute Node
Mesos Slave Process
Hadoop task-tracker
Task #1 Task #2
Container(Cgroups)
Task #3
![Page 35: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/35.jpg)
Mesos provides componentized resource
Mesos Slave Process
Mesos Containerizer
CGroups CPU isolator
CGroups Memory isolator
Launcher
Container foo
Task baz
Containerizer API
Executor bar
![Page 36: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/36.jpg)
Mesos provides pluggable resource isolation
External Containerizer
External Containerizer API
Mesos Slave Process
External Containerizer Program
Container foo
MySQL
Containerizer API
Ubuntu 13.10
Container bar
Ruby
Centos 6.4
github.com/mesosphere/deimos
![Page 37: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/37.jpg)
From Static Partitioning to Elastic Sharing
Static Partitioning
Elastic Sharing
WEB HADOOPCACHE
WASTED
FREEFREEHADOOP
WEB
CACHE
WASTED WASTED100% —
100% —
![Page 38: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/38.jpg)
Mesos has no single point of failure
Tasks keep running!
Framework
Masters
![Page 39: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/39.jpg)
Master node can fail-over
Tasks keep running!
Framework
Masters
![Page 40: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/40.jpg)
Slave processes can fail-over
Tasks keep running!
Compute Node
Mesos Slave Process
Mesos Executor Mesos Executor
![Page 41: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/41.jpg)
The UNIX Operating System Stack
SSHd
Linux, BSD
MySQLApache
Kernel
ApplicationsMemcached
Init, Upstart, Systemd Init System
![Page 42: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/42.jpg)
The Mesos Stack
Memcached
Mesos
RedisRails
Kernel
ApplicationsElasticsearch
Marathon Init System
![Page 43: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/43.jpg)
Mesosphere Demo
![Page 44: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/44.jpg)
Marathon
![Page 45: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/45.jpg)
What is Marathon?
“Init Daemon” for the data center
•Runs any Linux binary without modification (e.g. Rails, Tomcat, …)
•Cluster-wide process supervisor
!
Private PaaS
•Service discovery
•Automated software and hardware failure handling
•Deployment and scaling
![Page 46: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/46.jpg)
Marathon Design Goals
Simplify Fewer things to manage = fewer sources of error All machines have the same configuration AutomateNobody likes to get paged at nightAutomatically respond to hardware & software failures
Improve efficiency Humans are bad at estimating resource requirements Use software-controlled elastic resource sharing
Self-serve API Developers want to have control over their apps Give them direct access to cluster resources
![Page 47: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/47.jpg)
Marathon Key Features
REST/JSON API
Easy to use web interface
Authentication & SSL
Highly available - no single point of failure
Placement constraints (nodes, racks, etc.)
Service discovery & load balancing via HAProxy
Event system for integration with 3rd party components, like load balancers
!
![Page 48: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/48.jpg)
Marathon Workflow
3.foo.com1.foo.com 2.foo.com
Rails Rails
Mesos
Marathon
POST /v2/apps
![Page 49: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/49.jpg)
Marathon Workflow
1.foo.com 3.foo.com2.foo.com
Rails Rails
Mesos
Marathon
POST /v2/apps
Play
Play
![Page 50: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/50.jpg)
Marathon API - Launching Self-Contained Apps
•Command to start the app •URL(s) to the app archive/configuration •Environment variables
!POST /v2/apps { “id”: “Play”, “uris”: [“http://downloads.mesosphere.io/tutorials/PlayHello.zip”] “cmd”: “./Hello-*/bin/hello -Dhttp.port=$PORT”, “env”: {“SECRET”: “password123”} }
![Page 51: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/51.jpg)
Marathon API - Launching Dockers
•Starting with Mesos 0.19 containers are 1st class citizens •Deimos is the Docker containerizer !POST /v2/apps { “id”: “Cassandra”, “container”: { “image”: “docker:///mesosphere/cassandra:2.0.6”, “options”: [“-v”, “/mnt:/mnt:rw”, “-e”, “CLUSTER_NAME=prod”] } }
![Page 52: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/52.jpg)
Marathon API - Integration with Deimos
Mesos Master
Marathon
Launch 3x Redis
Mesos Slave
DeimosTask 1
Task 2docker run ...
Deimos
docker run ...
Mesos Slave
Deimos
docker run ...
Task 3
![Page 53: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/53.jpg)
Marathon API - Scaling Apps
•Just tell Marathon how many you want! !!
PATCH /v2/apps/Play { “instances”: 4 }
![Page 54: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/54.jpg)
![Page 55: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/55.jpg)
![Page 56: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/56.jpg)
![Page 57: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/57.jpg)
Marathon Service Discovery with HAProxy
Apps available on localhost & known port
HAProxy updates via Marathon REST API
HAProxy runs on every cluster node
Configurable policies
Rails
HAProxy
Rails
HAProxy
SQL
HAProxy
Search Search
HAProxy
Cache
Marathon
![Page 58: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/58.jpg)
Mesosphere Products & Services
Infrastructure • Try out Mesos on Amazon Web Services
https://elastic.mesosphere.io • On premise and cloud provisioning tools • Linux Packages • Mesosphere Plugins and Tools • Professional Services
• Training, Installation, Support Contracts
![Page 59: Apache Mesos and Mesosphere: Live webcast by CEO and Co-Founder Florian Leibert](https://reader033.fdocuments.us/reader033/viewer/2022060107/554a0398b4c905557a8b527a/html5/thumbnails/59.jpg)
Brand GuidelinesVersion 110.21.2013
Thank you.