HTTP API In Action by Max Neunhöffer, ArangoDB and Jörg Schad, Mesosphere

HTTP API in ActionMax Neunhöffer, ArangoDB

Jörg Schad, Mesosphere

© 2016 Mesosphere, Inc. All Rights Reserved. 2

Jörg SchadDistributed Systems Engineer,

Mesosphere

@joerg_schad

Max NeunhöfferSenior Software Architect

ArangoDB

@neunhoef


HTTP v1 APIs ● Mesos 1.0 Milestone ● Actually multiple APIs

○ Framework■ Scheduler■ Executor

○ Operator● RPC-based HTTP API● Language independent

schedulers!


In the beginning there was libmesos


The old days…

● Not portable● Hard to

debug● Upgrade

Dependency● No

Versioning

Scheduler

Java Library

NativeLibrary

mesos.jar libmesos.so

Mesos Master


Networking ● Container?● Firewall

Scheduler

Java Library

NativeLibrary

mesos.jar libmesos.so

Mesos Master


Let us break down walls….

© 2015 Mesosphere, Inc. All Rights Reserved.

Mesos APIs

8

Scheduler Master Agent Executor

Scheduler API Executor API

Framework API

Master API Agent API

Operator

Operator API

Internal API


But then business demanded

FAST DATALet us build something new!


API V1 Goals

● Work inside firewalls / containers● Allow pure language client libraries● API versioning● Backwards compatible API evolution● Well documented

FRAMEWORK PROTOCOL• Every call is a HTTP POST request

• Content-Type: application/json or application/x-protobuf

• Successful SUBSCRIBE call results in a “200 OK” streaming response

• Record-IO formatted events

• Persistent connection

• All successful non-SUBSCRIBE calls result in “202 Accepted”

•

WORKFLOW

Scheduler Master

SUBSCRIBE

POST /api/v1/scheduler HTTP/1.1

{ "type" : "SUBSCRIBE", "subscribe" : { "framework_info" : { "user" : "foo", "name" : "Example HTTP Framework" } }}

WORKFLOW

SUBSCRIBE Scheduler Master

WORKFLOW

HTTP/1.1 200 OK

Content-Type: application/jsonTransfer-Encoding: chunkedMesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af

<event length>{ "type": "SUBSCRIBED", "subscribed": { "framework_id": { "value": "12220-3440-12532-2345" }, "heartbeat_interval_seconds": 15 }}<more events><more events>

SUBSCRIBE

Streaming Response

MasterScheduler

WORKFLOW

Scheduler Master

Offers Event<event-length>{ "type" : "OFFERS", "offers" : [ { "offer_id":{"value": "12214-23523-O235235"}, "framework_id":{"value": "12124-235325-32425"}, "agent_id":{"value": "12325-23523-S23523"}, "hostname":"agent.host", "resources":[...], "attributes":[...], "executor_ids":[] } ]}

Agent

WORKFLOW

Scheduler Master

ACCEPT

POST /api/v1/scheduler HTTP/1.1

Mesos-Stream-Id: 130ae4e3-6b13-4ef4-baa9-9f2e85c3e9af

{ "framework_id" : {"value" : "12220-3440-12532-2345"}, "type" : "ACCEPT", "accept" : { "offer_ids" : [ {"value" : "12220-3440-12532-O12"}, {"value" : "12220-3440-12532-O12"} ], "operations" : [ {"type" : "LAUNCH", "launch" : {...}} ], "filters" : {...} }}

Agent

Scheduler Master

WORKFLOW

ACCEPT

ResponseHTTP/1.1 202 Accepted

Agent

WORKFLOW

Scheduler Master

ACCEPT

Launch Task

Agent

WORKFLOW

Scheduler Master

Launch Task

Executor

SUBSCRIBE

Agent

ACCEPT

SCHEDULER DISCONNECTIONS & PARTITIONS

• Master tracks the persistent subscription connection

• Reconnect within framework failover timeout

• Periodic HEARTBEAT events sent by master

Operator API

• Master and Agent API• Human Operator •• Event Stream• Avoid Polling /state_summary

• RecordIO format•

21

POST /api/v1/executor HTTP/1.1

Host: agenthost:5051Content-Type: application/jsonAccept: application/json

{ "type": "SUBSCRIBE",}

Operator API

• Master and Agent API• Human Operator •• Event Stream

• RecordIO format•

22

<event-length>{ "type": "TASK_UPDATED", "task_updated": { "task_id": { "value": "42154f1b-adcd-4421-bf13-8bd11adfafaf" }, "framework_id": { "value": "49154f1b-8cf6-4421-bf13-8bd11dccd1f1" }, "agent_id": { "value": "2915adf-8aff-4421-bf13-afdafaf1f1" }, "executor_id": { "value": "adfaf-adff-2421-bf13-adf23tafa21" }, "state" : "TASK_RUNNING" }}

Versioning

• Corresponding major versions

• API compatibility

• Protobuf like

http://localhost:5050/api/v1/scheduler

http://localhost:5051/api/v2/executor

API STATUS

• Scheduler API & Executor API : Stable

• Operator API : Experimental

• Client library implementations


● multimodel Database

● extensible


Writing a framework scheduler for a stateful service is hard:

● A part of the scheduling happens in the framework (accepting offers).● Getting the scheduling logic right is difficult● A framework needs to be resilient (use zookeeper)● The ArangoDB framework consists of 5140 SLOC in C++● It handles

● deployment,● dealing with persistent volumes, ● a part of the failover, and● a part of the up and down scaling.

ArangoDB Mesos Framework


libmesos makes it challenging to develop frameworks:

● The framework executable is usually deployed as Docker image.● There are two connections needed between framework and Mesos master in

different directions (network isolation problems).● One is limited to C++, Java or Python.● The libmesos source is huge: 190K SLOC● A Docker image to build a framework with libmesos needs >3GB● Documentation is mostly hidden in the source.● Debugging is a challenge (Docker, Mesos, binary messages, network issues).

Motivation ArangoDB


Solution: Use a SDK that works via the HTTP protocol.

The HTTP protocol provides the following advantages:

● any language works that can to HTTP/JSON● JSON is clear-text and easier to debug● using an SDK removes the amount of boilerplate code● do not need libmesos● much smaller footprint

Prototype: Mesos framework SDK for node.js

Still: scheduling and failure management

HTTP based ArangoDB

Writing Stateful Frameworks - Ken Sipe & Jörg SchadSaturday, 10:25

https://github.com/tobilg/mesos-framework


Challenges

● State Abstraction missing (MESOS-2916) ● Mesos-framework

● Persistent Volumes● Failover● Scaling● Operations (e.g., Updates)


● Shim● Switch between old/new API● Independent code

● Mesos-framework● JS based SDK● Not production!

● DC/OS SDK

How to get started?var scheduler = new Scheduler({

"masterUrl": "172.17.10.103", // If Mesos DNS is used this would be

"leader.mesos", otherwise use the actual IP address of the leading master

"port": 5050,

"frameworkName": "My first Command framework",

"logging": {

"level": "debug" // Set log Level to debug (default is info)

},

"restartStates": ["TASK_FAILED", "TASK_KILLED", "TASK_LOST",

"TASK_ERROR", "TASK_FINISHED"], // Overwrite the restartStates (by default,

TASK_FINISHED tasks are NOT restarted!)

"tasks": {

"sleepProcesses": {

"priority": 1,

"instances": 3,

"commandInfo": new Mesos.CommandInfo(

"sleep 10;"

),

"resources": {

"cpus": 0.2,

"mem": 128,

"ports": 1,

"disk": 0

}}}

Writing Stateful Frameworks - Ken Sipe & Jörg SchadSaturday, 10:25


● Use libraries● Shim for easy switching● Persistent Connection

● Connection: keep-alive● Transfer-Encoding: chunked

● HTTP Pipelining● React to heart beats● Implement authentication schema

Best Practices


Thank You!

Learn more by visiting dcos.io and arangodb.com.

HTTP API In Action by Max Neunhöffer, ArangoDB and Jörg Schad, Mesosphere

Technology

Transcript of HTTP API In Action by Max Neunhöffer, ArangoDB and Jörg Schad, Mesosphere