Post on 21-Jan-2018
DOING BIG DATA FORREAL WITH DOCKER
MESOSPHERE DCOSElizabeth Lingg
elizabeth@mesosphere.io
AGENDA1. Intro2. Mesosphere, Docker, and DCOS Overview3. Big Data Container Orchestration using DCOS and Docker4. Demo5. Q & A
INTROEngineering Manager @ MesosphereM.S. Computer Science with a Specialization in ArtificialIntelligence from StanfordB.S. Computer Science with a Minor in Math, B.S. Policyand Management from Carnegie MellonExperience in AI, Big Data, and SystemsEnjoys applying Distributed Systems to Manage andReason Over Large Amounts of Data
MESOSProvides primitives to author datacenter-native apps.
PRIMITIVES
Resources (cpu, mem, disk, ports)Asset fetchingTask state trackingAPI for the datacenter
STATUS QUO IS STATICPARTITIONING
AND USE OF VIRTUAL MACHINES
MESOS LET US TREAT A CLUSTER OFNODES...
AS ONE BIG COMPUTER
Not as individual
machines
Not as VMs
BUT AS COMPUTATIONALRESOURCES LIKE CORES, MEMORY,
DISKS, ETC.
WE LOVE CONTAINERS
MOST MODERN APPLICATIONS ARE A WEB OFCONTAINERS
A CONTAINER ORCHESTRATION PLATFORM
Containerization in Mesos, a brief history
MESOSPHERE DCOSSoftware to provide a complete OS: init, cron, apt-get,discovery, routingBeautiful web UI and CLISupportEcosystem of DCOS ServicesMesos Master and Mesos Workers Running in DockerContainers
DCOS UI
DCOS CLI$ dcos
Command line utility for the Mesosphere Datacenter OperatingSystem (DCOS). The Mesosphere DCOS is a distributed operatingsystem built around Apache Mesos. This utility provides toolsfor easy management of a DCOS installation.
Available DCOS commands:
config Get and set DCOS CLI configuration properties help Display command line usage information marathon Deploy and manage applications on the DCOS node Manage DCOS nodes package Install and manage DCOS software packages service Manage DCOS services task Manage DCOS tasks
BIG DATA DISTRIBUTEDAPPLICATIONS ON DCOS
Mesos Master and Mesos Workers Running in DockerContainersDistributed Applications Running in Containers on theMesos WorkersContainer Orchestration done by Apache MesosResource Allocation and Scaling Managed by ApacheMesos
BIG DATA DISTRIBUTEDAPPLICATIONS ON DCOS
Popular Distributed Apps easily deployed on a singleDCOS ClusterKafka, Cassandra, HDFS, Spark, and other Big DataServicesHealth checks and failure recovery are automated
APPLICATION NETWORKINGInteract with the CLI or REST API's to interact with theservicesMesos DNS ResolutionDocker Networking mainly done through host modenetworking, works seamlessly
DATA SECURITYServices storing secure data run on private worker nodesin the clusterPrivate nodes can only be accessed through VPNAs needed, services choose what is exposed through aproxy running on a public nodeDistributed Application can authenticate with the Masterusing Framework Authentication (Kerberos Option)
EXAMPLE: SIMPLE DOCKER APP ONDCOS
{ "id": "/mesosphere/cd-demo-app", "instances": 1, "cpus": 1, "mem": 512, "container": { "type": "DOCKER", "docker": { "image": "mesosphere/cd-demo-app:$tag", "network": "BRIDGE", "portMappings": [ { "servicePort": 28080, "containerPort": 80, "hostPort": 0, "protocol": "tcp" }<
EXAMPLE: CASSANDRA DCOSSERVICEFEATURES
Managed node configurationHealth MonitoringRest APIDNS Names for nodesMultiple Rings in one cluster
INSTALL$ dcos package install cassandra
CUSTOMIZABLE INSTALL OPTIONS{ "cassandra": { "cluster-name": "dev", "resources": { "cpus": 3.0, "mem": 6144, "disk": 30720 } }}
$ dcos package install cassandra --options=options.json
INSTALLING
HEALTHY
REST APIGET /node/all
GET /health/cluster/report
POST /node/{node}/replace
POST /cluster/repair/start
POST /scale/nodes?nodeCount={count}
DEMO!
Q & A
THANKS!LET'S CHAT!WE'RE HIRING!
DCOS: Join:
mesosphere.commesosphere.com/careers/