HTTP Plugin for MySQL!Ulf Wendel, MySQL/Oracle

The speaker says...

MySQL is the WebScaleSQL DBA darling:• The power and popularity of SQL• The – after all – deployment story• Scale-up: supports latest commodity hardware• Scale-out: asynchronous replication • Scale-out: in-memory, synchronous (2PC) replication• Scale-out: virtual synchronous, group based replication• Scale-out: sharding through Fabric

Just, we forgot about the web developer!

shell> curl --user user:password -X PUT -d $'{"msg": "Welcome, JavaScript user!"}' http://new.mysql.com/doc/pres/http/key_message

The speaker says...

Imagine, you are under such pressure that you have to revolutionize the way you work. Web developers do that. Infrastructure costs socialized through Open Source: web developers! Agile movement: web developers! Rusty database vendors don't deliver: Not Only SQL created!• Convenient APIs and relevant protocols• Really easy sharding• Schema flexibility and documents• Using their latest tools: JavaScript and JSON• No always failing short yet complicated ORM• No lobby standard ECISANO 5.7 Rev. 16 – buy now

Backend: It's LAMP. Period.Frontend: Dinosaur, it's 2014. Need a RESTful API. Backend: No time. And, no to framework Sendso...

The speaker says...

Every other day two developers clash on the question of databases... MySQL cannot ignore the needs of web developers. MySQL cannot hide itself behind the argument some third party framework could be used – that failed badly for replication. MySQL cannot hesitate and hope legacy code would stop migrations. Web developers completed two revolutions. Replacing MySQL would be a minor one. DBAs won't stop them.

What if MySQL speaks HTTP, stores JSON, returns JSON and preserves the richness of SQL?

Three HTTP APIsAll APIs return JSON

SQL

CRUD: flat JSON documents mapped to tables

Nested JSON documentsstored in tables

Key-Document semantics

The speaker says...

Every other day, a proxy is desired to make MySQL more accessible: HTTP – PHP|Perl|Python|... - MySQL. All the proxies translate HTTP to SQL. SQL lets them offer any API.

What if MySQL had a built-in HTTP to SQL proxy? Right from start, there could be a variety of APIs. Plain SQL over HTTP is a given. Everybody knows SQL, everybody understands the advantages of a declarative language, only SQL can expose the full feature set of MySQL. No attempt to dictate any whiteboard API: plain SQL. Google created AngularJS for CRUD applications. CRUD is a given. Nested JSON documents are a must.

http://labs.mysql.comFree (GPL), 80+ pages documentation, many examples

The speaker says...

The built-in MySQL HTTP APIs are real. You can get the HTTP Plugin for MySQL from http://labs.mysql.com (September 2014). As always, it's free and GPL. It comes with a 80+ pages PDF document that has many examples on using it. The HTTP Plugin is for the JavaScript users, for those fighting corporate firewalls that allow no other protocols but HTTP, those who love web services and those who are tired of writing yet another RESTful plain scripting language proxy for MySQL!

Next: Unboxing. Then a word on how it works, how it serves not only data but you. And finally an outlook.

UNBOXING: a seasonal scarf ready MySQL 5.7 Server

The speaker says...

Autum is coming to the northern hemisphere but before the nights get longer than the days, before we all start knitting scarfs, it's time for world's first unboxing.

In the box of the initial release is a special version of the MySQL Server. The server is based on the MySQL Server preview (5.7.5-m15) made available for MySQL Connect 2014. But, it has some additional feature tweaks. For now, you need this very server and no other.

Why that is the case will be explained in a minute.

UNBOXING: a high tech HTTP Plugin for MySQL

The speaker says...

MySQL stands for high tech: HTTP Plugin for MySQL! The treasure box Sakila (the MySQL mascot) accidently discovered at the beach contains the HTTP Plugin too.

One slide on installation to come, then we talk how it works.

UNBOXING: plain vanilla...

Install MySQL from the treasure box

Create a SQL user for the plugin• Set password, grant access to SQL object

Install the plugin• INSTALL PLUGIN myhttp SONAME 'libmyhttp.so'

Edit my.cnf• Set default SQL user for plugin, HTTP basic auth, port, etc.

Restart MySQL• If you edit first, then load – recall to use loose option prefix

The speaker says...

The installation procedure is not different from any other plugin. Some stuff the plugin needs to work is to be set up first. The HTTP plugin runs SQL and does that – by default – as a certain user. Create that user. Personally, I prefer loading the plugin in the next step before making changes to the MySQL Server configuration file. This allows me not to bother about a server that barks about unkown settings. Once installed, the plugin begins listening to HTTP requests (default 8080). If you have a server running on that port, edit my.cnf first to set port, user etc., then start MySQL. Recall to use the „loose“ prefix trick. Then install the Plugin. The manual has the details... We get back to settings in the usage section after: how it works (now).

How? Extending MySQL!

MySQL

Storage Engine Plugin

Core

Deamon Plugin

Deamon Plugin

I_S Plugin

Full-Text Parser Plugin

Semisync Replication Plugin

Audit Plugin

User-defined functions

Authentication Plugin

Password-Validation Plugin Protocol Trace Plugin

Plugin Services

The speaker says...

The MySQL manual spends 80 pages educating readers how to extend MySQL: there are user-defined functions (UDF) and there are plugins. SQL can be extended with UDFs. Nine different types of plugins exist. Best known should be the storage engine plugins (InnoDB, Memory, NDB, …, 3rd party) and the authentication plugins (MySQL native, SHA256, …). The HTTP Plugin is a daemon plugin. A daemon plugins is a do-as-you-like library that is loaded into the MySQL Server. Plugins have access to half a dozen plugin service provided by the server (Memory, thread management, ...). The HTTP Plugin is build atop of new plugin services.

The Protocol deamon plugins

MySQL

HTTP Plugin

Daemon Plugins

Memcache PluginCore

MySQL ConnectorMemcache driverHTTP Client

HTTP Protocol

(JSON)

Memcache Protocol

(Binary)

MySQL Client/ServerProtocol

(Binary)

Deamon plugins can start a network server and listen to an arbitrary protocols to serve data. In 2009, the year the equation NoSQL = Not Only SQL was proposed, the first community plugin did exactly that. For rich queries the customers used SQL, for fast but simple queries a home-brewed daemon plugin using lower level APIs. MySQL was at the forefront but few know it.The MySQL Server comes with a free Memcache Plugin for InnoDB since 5.6, so does MySQL Cluster. The customers and community desire was to kill any read access overhead: SQL parsing, network traffic. For write they desired ACID, in-place update and predictable performance under all circumstances.

The speaker says...

The query language flexibility

MySQL

HTTP Plugin

Daemon Plugins

Memcache PluginCore

MySQL ConnectorMemcache driverHTTP Client

SQLKey-Document Key-Value

SQL

The HTTP Plugin Lab Release is not designed for maximum speed. It targets the JavaScript user. Today, it is just a demo of what APIs we could offer to the JavaScript user. The Lab Release does nothing you cannot do yourself for years already: all HTTP requests are mapped to plain SQL internally. But it does it for you. The day the APIs have been finalized we can boost the performance.

We are also exposing the richness of SQL. There is no we know the best ORM and you must love it. Did they ever work? Use SQL or try the CRUD interface we sketched or store nested JSON documents. At this point, all we do is add a HTTP to SQL proxy to MySQL...

The speaker says...

Your trouble vs. our trouble

MySQL

HTTP Plugin

HTTP Client

SQL, Key-Documentover HTTP

HTTP Client

Some serviceover HTTP

MySQL Connector

Webserver

Script Language

MySQL

This Lab Release is no rocket science. Effectively, we are only announcing the intend to offer an HTTP interface as a convenience feature. We move the proxy you write today to make MySQL accessible through HTTP into the server – as an optional plugin. We take the trouble of thinking about security or performance. There is one drawback, assuming the APIs work for you. Our solution could be a little less scalable. A shard nothing architecture, an echo of the Unix way of having small independent programs, is often the best way to scale. But then, the external proxy has more connections and may be slower... Wait for it to stabilize before you stress test – not worth checking today. Let's talk using and APIs.

The speaker says...

http://new.mysql.com/sql/pres/SELECT%20'Hello World!'http://new.mysql.com/crud/pres/http/hellohttp://new.mysql.com/doc/pres/http/hello

There are three APIs with some properties in common:• they have JSON all over• guess what: they all use HTTP, which sets limits• the same (yet unsecure) security pattern• they share the same restrictions

The APIs differ by:• their query language• their current interest in REST...

Common things first at the example of the SQL API.

The speaker says...

The SQL endpoint and JSONshell> curl --user basic_auth_user:basic_auth_passwd

--url "http://127.0.0.1:8080/sql/db/SELECT+1"

[

{

"meta":[

{"type":8,"catalog":"def","database":"","table":"",

"org_table":"","column":"1","org_column":"","charset":63,

"length":1,"flags":129,"decimals":0}

],

"data":[

["1"]

],

"status":[{"server_status":2,"warning_count":0}]

}

]

The SQL endpoint is our Connector/JavaScript. What you see is exactly what a standard MySQL Connector gets in reply to the query. But the standard MySQL Connector is using the binary MySQL Client/Server Protocol and all data would look funny. Because the same information is returned the HTTP client is not feature constrained. That is a sweet lie as we will see later but you get the idea. The HTTP client, the JavaScript user is not a second class client who is limited to restricted APIs up-front.All JSON is UTF-8. For now, the HTTP Plugins' web is UTF-8. Binary data and funny german umlauts are encoded using \uxxxx notation, as json.org & friends ask for.

The speaker says...

No session concept for the 1.0• Stateless helps with scaling and reduces complexity

• Some feature, e.g. SQL session variables, have limited value

ACID but autocommit• Each web request is a transaction

• Difficult to isolate multi-statement/business transactions

Dot zero: HTTP is stateless

shell> curl --user basic_auth_user:basic_auth_passwd

--url "http://new.mysql.com/sql/db/SET+@no='session'"

{ "server_status": 2, "warning_count": 0, "affected_rows": 0, "last_insert_id": 0 }

The stateless nature of HTTP is no natural match for SQL. ACID transactions are an integral part of SQL systems. Transactions are units of work made of one or more statements. The SQL endpoint lets you carry out one SQL statement per request. Every new HTTP request is independent of the previous one, thus every statement forms its own transaction: it runs in autocommit mode. You get ACID properties: atomicy, consistency, isolation and durability. E.g. two concurrently executed web requests are properly isolated from each other.

Sessions? Makes a lot sense to me! Not only for multi-statement/business transactions.

The speaker says...

Dot zero: user/security concept

HTTP Client

No SSL

Preconfigured user

Authentication

HTTP Plugin

MySQL

HTTP Client

Request user

Authentication

HTTP Plugin

MySQL

SSL

SQL grants

The user/security concept is two-staged. First, a HTTP client authenticates itself towards the HTTP Plugin. Then, the HTTP Plugin executes SQL statements on behalf of the HTTP client as a certain SQL user. If no SSL is used, the HTTP Plugin logins in towards MySQL as a preconfigured SQL user. The HTTP world and the SQL world are decoupled. With secure connections (SSL) the user name and password obtained from the HTTP world can be passed to MySQL as login credentials.

A HTTP client is just another MySQL client. Use the SQL permission system to restrict access!

The speaker says...

HACKER Welcome!MySQL morons using HTTP Basic Authentication...

The initial dot zero Lab Release is using HTTP Basic Authentication all over. All resources – SSL and Non-SSL – are protected with basic auth.

HTTP Basic Authentication is fast to implement. Everybody understands the examples because of their simplicity. That is perfect for a /preview/, that is perfect to kick of a discussion.

Unfortunately HTTP Basic Authentication is one of the least secure choices one can make.

The speaker says...

SSL: User/security concept

HTTP Client

OpenID, OAuth...

HTTP Plugin

MySQL

SSL

HTTP Client

Authorization

Password exchange

HTTP Plugin

MySQL

SSL

How hard can it be to support any proper HTTP authentication method: a day for OAuth, two weeks for OpenID? The Lab Release uses HTTP Basic Auth as a vehicle that combines two steps, which is confusing but was fast to implement. For SSL connections, and only SSL connections, users should first authenticate themselves through some proper HTTP authentication method. Once authenticated, they shall be able to login as a MySQL user of their choice. We still need to fiddle out the details and we lack something here as the outlook will show.

Next common properties of all APIs: restrictions.

The speaker says...

Dot zero: restrictions

No query cache support• It's HTTP – use a standard HTTP caching!

• MySQL Support often suggests disabling the query cache

No commercial thread pool plugin support• There are now HTTP network threads and SQL worker threads

• HTTP layer can do multiplexing

Not all MySQL pluggable auth methods supported• Some pluggable auth methods have handshakes

• Handshake procedures require use of MySQL networking layers

The preview has some restrictions. They all boil down to the fact that building a proxy into MySQL is an all new requirement. Various modules in MySQL have not been designed for this new use case. How could they? It is a new use case...

Do you mind being restricted to use MySQL SQL accounts that have been created with MySQL's default authentication plugin? Do you mind following a MySQL Support advice of not using the query cache? Recall, that you should deploy InnoDB with 80% of your memory dedicated to the buffer cache. Disk access is rare.

The speaker says...

Unlimited: all SQL statements• DDL, DML and utility statements

• Multi-resultset statements supported, e.g stored procedures

Unlimited: everything the server returns• Meta data, query data, server status

• Freedom to build your own Connector/JavaScript

The SQL endpoint

[{

"meta" : [ … ],

"data" : [ … ],

"status" : [ … ]

}]

First class customers use the SQL endpoint. It has the most to offer. There are no restrictions on which SQL statements can be executed. Use the MySQL SQL permission system to set any, if needed. No limited white-board API is imposed on you.

All the information the server returns in reply to a query is converted to JSON and sent to the client. A standard MySQL Connectors using the MySQL Client/Server Protocol gets the same information but in binary form. Because the same data is delivered, clients can develop their own Connectors, their own higher level APIs and data structures.

The speaker says...

Unlimited result set meta data[{

"meta":[{

"type" : 4,

"catalog" : "def", "database" : "pres",

"table" : "sql_types", "org_table" : "sql_types",

"column" : "col_float", "org_column": "col_float",

"charset" : 63,

"length" : 12,

"flags" : 4097,

"decimals": 31

}],

"data" :[ … ],

"status":[ … ]

}]

On the slide is the result set meta data for SELECT col_float FROM sql_types. The full table definition of the example table sql_types is given in the manual, however, col_float is of the SQL type FLOAT.

For every column in the result set you get one object in the meta array. Every object has the same members: there's the database name, the table name, the column name, the charset and so forth.

Why all this result set meta data?

The speaker says...

Types: string or NULLshell> php -r ' $m = new mysqli("host", "user", "password", "db", 3306); $m->query("SELECT col_float FROM sql_types")->fetch_assoc());'

array(1) {

["col_float"]=>

string(6) "0.9999"

}

shell> curl --user basic_auth_user:basic_auth_passwd --url "http://…/sql/db/SELECT+col_float+FROM+sql_types"

[{

"meta":[ … ],

"data":[ ["0.9999"], … ],

"status":[ … ]

}]

The SQL endpoint converts all SQL result data to JSON strings or JSON NULL.

A conversion to string is the servers standard behaviour for non-prepared statements in the MySQL Client/Server Protocol too. This server implementation detail still shows in PHP. Still today, PHP returns numbers as strings. That made a lot sense in the last millennium. HTTP GET and POST data are strings. PHPs' dynamic typing was a relief for those doing web development in C before.

There's still a point for sending strings today.

The speaker says...

No type mapping, no loss

SQL type JSON candidate Information lossSMALLINT Number Precision, rangeFLOAT Number Precision, rangeVARCHAR String Precision, charsetDATE String Type, rangeBOOLEAN True, falseNULL Null

How would the server know who consumes the data• Which rules?

• Information loss allowed?

The server does not know the type system of the client. If, for example, SQL data types had been mapped to JSON data types and no meta data had been included, information had been lost. Complicated? Choose between a Bugatti Veyron and a Suzuki Maruti 800. The Suzuki has simpler controls. The Suzuki costs less. The Suzuki is more fuel efficient. All speaks for the Suzuki but we won't give it to you. The SQL endpoint is as luxury and as complex as the Bugatti. Here's why: it's easy to use a hammer to shrink the Veyron to the size of the Suzuki. It's hard to make the Maruti go 400 km/h. Even when dropping from the sky – cW too high ;-).

The speaker says...

JSON with padding• Optional escaping into string

• CORS on the radar

JavaScript same origin policy

var ret = $.ajax({

url: ".../sql/db/SELECT%20'Hello world!'%20FROM%20DUAL",

type: "GET",

username: "basic_auth_user", password: "basic_auth_passwd",

dataType : "jsonp", jsonp: "jsonp",

success: function( json ) {

alert("First row: " + json[0].data[0]);

},

});

Browserside JavaScript has two options to overcome the same origin policy: JSON with padding (JSONP) and Cross-Origin Resource Sharding (CORS). JSONP is there for the Lab Release, CORS is missing. Add the GET parameter jsonp=<my_callback_name> to request to get JSONP replies. The result will be: my_callback_name(<json>). The slide shows an example for jQuery.

Optionally, you may request the <json> to be returned as a string value, like so: my_callback_value("<json>"). If you need the latter, add the GET parameter jsonp_escape to the request URL.

The speaker says...

HACKERs' heaven!Popcorn for everbody. MySQL in the web...

Ever noticed NoSQL systems offering HTTP interfaces? Peace: it is an optional plugin...! You decide whether you load it into MySQL or not.

The HTTP Plugin currently lacks settings for disabling endpoints. Accidently exposing sensitive data, for example, because of inappropriate SQL permissions, is a security risk. We will talk about that at the end. The SQL endpoint is the unlimited first class customer API. Great power comes with great responsibility. Use it with greatest care!

Next: the more restricted CRUD endpoint!

The speaker says...

Flat JSON objects mapped to tables• Always JSON objects towards the caller

Limited query language • Data manipulation only: GET, PUT, DELETE

• Key-document semantics

• No data definition statements

• No administrative statements

Less network traffic• No result set meta data

The CRUD endpoint

The CRUD endpoint features simple key-document semantics for unnested JSON documents. Only data manipulation operations can be performed. Whereas the SQL endpoint allows no other HTTP methods but GET, the CRUD endpoint accepts GET, PUT and DELETE requests. In that sense it shows more love for REST.

On the outside there are JSON documents (objects) only. On the inside all requests are mapped to SQL in a straightforward way. Let's see how.

The speaker says...

Document = SQL table row• Documents are flat, they are limited to scalar types

• Underlying tables must have a single column primary key

The CRUD SQL mapping

CREATE TABLE `simple` (

`id` int(11) NOT NULL AUTO_INCREMENT,

`col_a` varchar(255) DEFAULT NULL,

PRIMARY KEY (`id`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8

shell> curl --user basic_auth_user:basic_auth_passwd

--url "http://.../crud/db/simple/1"

{"id":"1","col_a":"Hello"}

It is that simple: SQL table rows are returned as JSON objects. Plain SQL tables serve as JSON document containers.

Each document is derived from only table only. Because MySQL SQL tables support scalar types only, the corresponding JSON douments contain scalar types only such as JSON string, number, boolean or null. The CRUD endpoint does not support JSON documents containing arrays or objects.

Every mapped table must have a single column primary key. The primary key is used for the key-document access logic.

The speaker says...

GET = SELECTshell> curl … --url "http://../crud/db/sql_types/1"

{

"id" : "1", "col_char" : "CHAR(127)",

"col_null" : null,

"col_date" : "2014-08-21",

"col_decimal": "123.45",

"col_float" : "0.9999",

"col_bigint" : "9223372036854775807"

}

shell> curl … --url 'http://../crud/db/simple/'

{

"errno" : 2000,

"error" : "The request URL must include a primary key value"

}

The initial version features key-document accesss only. Therefore, all GET requests must include a primary key value and follow the URL pattern protocol://server:port/crud/database/table/pk_value.

Like the SQL endpoint, the CRUD endpoint returns all data either as a string or null. All the APIs are far from finalized. A possible future optimization would be a better mapping between SQL and JSON data types. For example, boolean values could be returned as true/false JSON constants.

The speaker says...

Key-document only? No :-) !

MySQL

HTTP Plugin Memcache Plugin Core

MySQL ConnectorMemcache driver

JS in browser

Rich SQLTuneable K/V Fast K/V

Rich SQL

Standard client, e.g. PHP

JS framework

MySQL 5.7.5 handles up to ~625.000 point select SQL queries and ~1.100.000 comparable Memcache accesses per second. This clearly shows the tuning potential behind simplified key-value (document) style APIs. Standard MySQL clients need two drivers to profit from the rich query capabilities of SQL and the performance of K/V. The CRUD endpoint is untuned. First, there needs to be a proper API (not the current...), then one can optimize. Once optimized HTTP clients can choose by just changing the request URL slightly: sql/ vs. crud/ respectively doc/. Could not be better, could it?

Back to the CRUD endpoint basics: HTTP PUT

The speaker says...

PUT = REPLACEshell> curl … -X PUT -d '{"col_a":"Ahoy"}' --url "http://.../crud/db/simple/1"

{

"affected_rows": 2,

"warning_count": 0

}

shell> curl … --url "http://.../crud/db/simple/1"

{

"id" : "1",

"col_a": "Ahoy"

}

shell> curl … -i -X PUT -d ''

--url "http://.../crud/db/simple/1"

HTTP/1.1 400 Bad Request…

HTTP PUT is mapped to SQL's REPLACE statement in the CRUD endpoint of the initial Lab Release. PUT expects a JSON object as input. The object must have one top level member for each column in the underlying SQL table that expects a value. If you fail to provide proper input, the HTTP Plugin for MySQL barks at you: 400 Bad Request.

No attempt is being made to hide the SQL reply to the REPLACE statement. It is one of the many edges we failed to cut off for the development announcement release. Another edge is the lack of clever mapping like PUT = INSERT, PATCH = UPDATE. Adding that is about as hard for us as it is for you in your HTTP to MySQL proxy...

The speaker says...

DELETE = DELETE, that's easy!shell> curl -i … --url "http://.../crud/db/simple/1"

HTTP/1.1 200 OK

…

{"id":"1","col_a":"Ahoy"}

shell> curl -i -X DELETE … --url "http://.../crud/db/simple/1"

HTTP/1.1 200 OK

shell> curl -i … --url "http://.../crud/db/simple/1"

HTTP/1.1 404 Not Found

…

{"errno": 2000, "error":"Not found"}

HTTP DELETE is translated to SQL DELETE. That's it.

There is some support for HTTP principles. For example, HTTP methods are mapped appropriately to data manipulation commands. Even HTTP headers are set: 200 OK, 400 Bad Request, 404 Not Found and so forth. This alone does not make a REST API and ticking of the checklist for the buzzword is the least (important).

Next: nested JSON documents – SQL completely hidden.

The speaker says...

Nested JSON objects mapped to tables• Always JSON objects towards the caller

Limited query language • Key-document semantics

• Data manipulation: GET, PUT, DELETE

• Data definition: Table creation and removal

• Optimistic locking

No sign of SQL• No cryptic affected_rows ...

• Leave room for possible optimizations

The DOCUMENT endpoint

When it comes to HTTP details and access methods, CRUD and DOCUMENT are very much the same. DOCUMENT is just a tiny bit more.

If you consider the design space for possible APIs of a HTTP Plugin that wants to speak JSON all over, you may notice that the two extremes are plain SQL and a JSON document store. In our initial Lab Release, the CRUD endpoint sits in the between the two extremes. The DOCUMENT endpoint is the „what if MySQL bend itself towards a JSON document store“. Stay seated: an API does not make MySQL a proper document store!

The speaker says...

JSON goes into a BLOBshell> curl -i -X PUT --user …

--url "http://.../doc/myhttp/new_table/"

HTTP/1.1 201 Created

…

{"info": "Table created"}

mysql> SHOW CREATE TABLE new_table\G

*************************** 1. row ***************************

Table: new_table

Create Table: CREATE TABLE `new_table` (

`_id` varchar(36) NOT NULL,

`_rev` bigint(20) unsigned NOT NULL DEFAULT '0',

`_extra` blob NOT NULL,

PRIMARY KEY (`_id`,`_rev`)

) ENGINE=InnoDB DEFAULT CHARSET=utf8

Use HTTP PUT requests to create tables for storing documents, to insert documents and to update them. There are no restrictions on what kind of JSON objects the endpoint handles. The API accepts every valid JSON object. Internally, JSON documents are stored in a BLOB column of a standard SQL table.

The primary key of the underlying SQL table is made of two columns: the id or key that appears in request URLs and a revision counter. The revision counter is managed by the HTTP plugin to implement optimistic locking.

The speaker says...

Optimistic locking

MySQLHTTP ClientHTTP Client

GET ../key

{"_rev":1, …}

{"_rev":"1", …}

PUT ../key OK: _rev: 2{"_rev":1, …}

PUT ../key Conflict!…1}

Arguing that HTTP is stateless and reflecting that fact in the APIs is great if you are short on time. But then, without session support, without multi-statement transactions – all endpoints use autocommit – how can two clients avoid and detect conflicting wirtes? One can leave the task to the client, you the developer. One can consider multi-versioning. This raises at least a garbage collection problem because the server cannot know when old revisions may be removed. Or, the DOCUMENT endpoint supports optimistic locking – which just works if the clients cooperate. Means: a client must not modify the special JSON document object member _rev.

The speaker says...

Solve conflicts by refetchingshell> curl --user … --url ".../first_message"

{"_id":"first_message","_rev":1,"msg": "No big news" }

shell> curl -X PUT -d '{"_id": "first_message","_rev": 1,"msg": "Optimist?"}'

--user … --url ".../first_message"

{"info": "Document updated"}

shell> curl -X PUT -d '{"_id":"first_message","_rev":1,"msg": "Optimist?"}' --user … --url ".../first_message"

{"errno": 2000, "error": "Update failed. Your revision does not match the current revision"}

Here's an example of optimistic locking in action. Every JSON document managed by the DOCUMENT has two special members: _id and _rev. _id is the key of the document. _rev is the revision number used for optimistic locking. When a JSON document is written back to the database, the database compares its revision number with the revision stored in the database. Should they be equal, the write is accepted and the database increments the revision. If not, then someone must have updated the document already and the write is rejected. In case of a conflict, a client should fetch the latest version, merge the changes to solve the conflict and try again.

The speaker says...

PUT = CREATE TABLE, REPLACEshell> curl -i -X PUT

--user … --url "http://.../doc/db/another_table";

HTTP/1.1 201 Created

{"info": "Table created"}

shell> curl -i -X PUT -d '{"words": ["Hello", "world"]}'

--user … --url "http://.../doc/db/another_table/key"

HTTP/1.1 200 OK

{"info": "Document added"}

shell> curl -i -X PUT -d 'is this json?' --user … --url "http://.../doc/myhttp/another_table/key"

HTTP/1.1 400 Bad Request

{"errno":2000, "error":"Invalid JSON"}

For the REST, let's do a walkthrough the HTTP methods supported. In the initial development release, the DOCUMENT endpoint has slightly more features than the CRUD endpoint. The CRUD endpoint will certainly be updated. Not only AngularJS framework lovers shall have free choice between the two types of mapping from JSON to SQL, both APIs shall offer comparable feature sets.

The DOCUMENT endpoint supports PUT for creating new tables, inserting nested JSON documents into them and updating documents. Nothing fancy but the optimistic locking. Invalid JSON is rejected.

The speaker says...

GET = SELECTshell> curl --user …

--url "http://.../doc/db/another_table/key"

{ "_id": "key", "_rev": "1", "words": ["Hello", "world"] }

shell> curl --user … --url "http://.../doc/db/another_table/"

{

"another_table": [

{ "_id": "key","_rev": "1","words": ["Hello", "world"] }

]

}

shell> curl -i --user … --url "http://.../doc/db/another_table"

HTTP/1.1 400 Bad Request

A HTTP GET request against the DOCUMENT endpoint either returns one document identified by its key (id) or all documents from a table.

Recall, that the access method is key-document only. For now, if you need filtering and the like, switch to the SQL endpoint. Change the request URL and use SQL for filtering.

The speaker says...

Filtering? Indexing? Coming...

See labs.mysql.com• JSON UDFs, work in progress: more functions than listed below

• MySQL 5.7.4: Virtual columns, stay tuned on functional index

Search• By value: JSON_SEARCH()

• By key: JSON_CONTAINS_KEY(), JSON_EXTRACT()

Modify• By value: JSON_MERGE()

• By key: JSON_APPEND|REMOVE|REPLACE|SET()

With MySQL 5.7.4 it is finally possible to work on JSON using SQL. The release brings:• updated SQL functions for working on JSON• virtual columns• … and stay tuned on function based index

The SQL functions let you search JSON and modify it. They fail to close the gap between the JSON and SQL data models but if you stay within the JSON model it works. The HTTP Plugin means JSON all over... Also, indexing JSON will boost search beyond key-document.

The speaker says...

DELETE = DELETEshell> curl -X DELETE --user …

--url "http://.../doc/myhttp/another_table/key"

{

"info": "Document removed"

}

shell> curl -X DELETE --user … --url "http://.../doc/myhttp/another_table/"

{

"info": "Table dropped"

}

shell> curl -X DELETE -i --user … --url "http://.../doc/myhttp/another_table/"

HTTP/1.1 404 Not Found

To remove documents and tables issue HTTP DELETE requests. That's it – very similar to the CRUD endpoint.

The speaker says...

Dear JavaScript users, please check the manual.Some examples already there, more coming.

RecapHTTP Protocolin addition to

MySQL Protocol

JSON all over

Key-Document and SQL complementing each other

SQL support for JSONgetting 'real' with

MySQL 5.7.4+

Hmm...? What's Sakila doing...

Sakila, you put it back in?

Oh, no... back to the sea ?!

Hidden treasures? Labs leaks?

Yammy!

A MySQL Lab Releases can be anything from almost production ready to a development announcement. For example, there have been several prior releases of the SQL JSON functions, which means they should work reasonably well. The day MySQL 5.7.4 was released a colleguage was praised for contributing to the SQL standard around computed columns, which means they should work reasonably well...

The HTTP Plugin 1.0.0 is a yammy development announcement release. It's all about discussing it early. That's why we have 80+ pages documentation from day one but edges in the APIs.

The speaker says...

Leaving the safe harbourGlancing over the ocean of possibilities

More open talk: ideas.

If you are a developer using MySQL, you may stop here. The presentation has explained what the HTTP Plugin is, why it was created, what it has to offer and what the current status is.

What's left is a deeper look inside, how it's done inside MySQL. That's for MySQL experts who seeks to understand MySQL internals. Understanding them is a precondition to use them in a different context, tailored to the own needs.

The speaker says...

Module AModule B

Module C

Module D

Plugin API

The challenge in doing a HTTP Plugin is not the HTTP API itself. How hard can it be to add something developers do every day to MySQL? How hard can it be to add a Proxy to MySQL?

Some say MySQL is not modular. Some say MySQL lacks clearly seperated modules in the code. Is MySQL a ball of woll with knotted fiber? Some say the code base shows its age. Is MySQL in the autum of its life, does it need a scarf? Well, RDBMS have undergone permanent refactoring in the past 50 years and they are sill around. NoSQL has its first wave yet to come. Think MongoDB and storage.

(Insider: Steward, I took that photo prior to your blog post.)

The speaker says...

Module AModule B

My new API :-)

The problem is a different one. With every new feature request someone comes and invents a new module. Some say, the authors of the HTTP Plugin invent new non-existing modules every other day.

There are two new plugin services inside the HTTP Plugin. Let's discover their potential beyond the HTTP Plugin.

The speaker says...

Plugin Services

MySQL

HTTP Plugin

HTTP Client

SQL Execution Service Authentication Service

SQL or any derived query method

Output Formatter Web Server, ...

The HTTP Plugin is build atop of two plugin service. The first is an authentication service to log in a MySQL internal SQL execution thread as a certain MySQL user. Once authenticated and authorized, the MySQL internal SQL execution thread uses the SQL execution service.

The results are returned as a binary string stream. The stream is processed by an output formatter, which is provided by the plugin calling the service. The plugins output formatter can serialize query results into any format. In case of the HTTP Plugin, its JSON. Could have made it BSON but which JavaScript client wants that...

The speaker says...

Proxy: SQL over Protocol X

MySQL

HTTP Plugin

HTTP Client

SQL

JSON Serialization

SQL Execution Service

Websocket Client

Websocket Plugin

JSON Serialization

SQL

Your Client

Your network Prot.

Your serialization

SQL

The HTTP Plugin is a protocol and data serialization format SQL proxy built-in to MySQL. It is one that is tailored for the JavaScript user. This restricts the protocol options to HTTP and Websocket.

In 2009 some MySQL high-end community users added an alternative, faster communication channel to MySQL for key-value style queries. Many expert users restrict their developers to a subset of SQL for performance reasons. The MySQL Client/Server Protocol has to support all SQL features and all programming language communities. Any ideas for optimized SQL centric protocols tailored to your users?

The speaker says...

Proxy: for data models

MySQL

HTTP Plugin

HTTP Client

JSON Document

SQL Rewrite Schema Mapping Util: e.g Catalog

Relations

The HTTP Plugin is data model proxy. Closing the gaps between data models can be done in many ways. In its early days, when MySQL was still looking for its space in the market, the storage engine API was used to offer MySQL as SQL frontend for about anything. Here is a suggestion from the Connectors team! In a perfect world, the Connectors team understands developers and their motivation. Working at this level is easier, more can contribute. The RDBMS world tried data model mappings in the past: it was never fast. But, the systems where never fast to support new models either!Getting it right is hard. Time to market counts. Web developer = no mercy = they do revolutions...

The speaker says...

Proxy: simplified shardig

Any shard

HTTP Plugin

HTTP Client

Key-document

SQL Execution

Target shard

301 Moved

FabricWhich shard?

MySQL supports auto-everything sharding since a decade. It comes with MySQL Cluster. MySQL Cluster just learned active-active asynchronous replication with automatic conflict solution between synchronous clusters in different data centers! Problem is, the web user wants InnoDB. So we build MySQL Fabric. Fabric manages farms of MySQL servers and lets you shard data across them. Problem is, it is not transparent and requires special clients. Imagine you limit the scope to stateless HTTP and key document. Stateless means redirect is no pain. HTTP means you can use slim clients and 301. Document means cross-shard joins can be avoided... Extra hop? Pffft, look at MySQL Cluster.

The speaker says...

Let's not loose focus: JS! V8!

MySQL

HTTP Plugin

HTTP Client

Improved security, more freedom

app/ - v8 JavaScript sql/, crud/, docs/ - built-in

SQL Execution Service Authentication Service

Let's not loose focus! The HTTP Plugin is for the developer not the DBA and database architect. That's why MySQL 5.7 brings true multi-master replication – its ease of use. Multi-master has scalability limits (~10...20 machines) but how many web developers are happy with a single of todays monster machines (6k Euro price point = 256GB RAM, 7TB disk, …) and bother only about dev features? The HTTP Plugin is specifically for the web and JavaScript developer. What could be more desired than adding the v8 JavaScript engine to the plugin. This also enables scripts to implement additional security rules. Rules that go beyond what the MySQL user system has to offer!

The speaker says...

A world of optionsMake it happen, today!

It is a world of options. No decisions have been made about any of them. Looking forward to hear back or see you implement your ideas for MySQL.

The speaker says...

THE END

Contact: ulf.wendel@oracle.com

The speaker says...

Thank you for your attendance!

Upcoming shows:

International PHP Conference (Munich),

October 26 - 29, 2014

Your place? Just ask!