Improving sanity with structured binary formats and protocols
-
Upload
kyle-drake -
Category
Technology
-
view
6.530 -
download
2
description
Transcript of Improving sanity with structured binary formats and protocols
Learning how to let go:Improving sanity with structured binary data-interchange formats and protocols
Kyle DrakeNet Brew Ventures
If you’re doing a lot of text-based serialized object passing, and want a way to improve serialization time and payload size, try a binary format.
Why• Smaller size, representing the same
information (esp. vs XML)
• Usually faster for CPU as a consequence
• Surprisingly easy to implement (with help)
• Lots of well-tested, streamlined solutions
Why Not• No quick language support (if custom)
• Hard to understand without good docs
• Potentially harder to debug low-level if something bad happens (data entanglement/corruption?)
• Binary != Performance (Measure!)
Two Approaches• Custom, hand-rolled format you write
• Frameworks (Protocol Buffers, MessagePack, BERT, others)
Custom Binary Format• This is often a bad idea.
• No, seriously.
• Requires a very compelling reason IMHO.
Let’s look at a poorly designed custom binary format, via: The Apple Push Notification Service.
Wait, why not just use JSON for the whole thing then?
Success: no responseError: socket disconnectionInterim: unreported data loss
W.T.F.
ROFLSCALE TIPS FTW
My solution:
Meanwhile, at Google:
I think this crap is why people hate working with binary so much.But there is a better way.
Let’s look at some structured ways to work with binary!
Protocol Buffers• Developed by Google
• “3 to 10 times smaller, 20 to 100 times faster than XML”
• Requires a pre-defined .proto file
• In effect, it has a language agnostic “schema”.
Protocol Buffers
Protocol Buffers
Protocol BuffersI’m doing work with the Bitcoin protocol right now. I really wish it used protocol buffers. Not just for size, but for safety and ease of use.
Bitcoin lead core developer is warm to idea:https://bitcointalk.org/index.php?topic=632.msg6656#msg6656
Let’s look at some “schemaless” structured binary formats.
JSON• Number (50, 2.33)
• String “howdy”
• Boolean (true or false)
• Array [1,2,3]
• Object {“key”: “value”}
• null (empty)
MessagePackBasically the same thing.
Except faster and smaller!
Super simple:
SOURCE: INDIEGAMR.COMHTTP://INDIEGAMR.COM/CUT-YOUR-DATA-EXCHANGE-TRAFFIC-BY-UP-TO-50-WITH-ONE-LINE-OF-CODE-MSGPACK-VS-JSON/
Painless API Integration
• /route?format=msgpack
• Content-Type: application/x-msgpack
• Accept: application/x-msgpack
• Plugs right in, if you’re using JSON.
HAVE A JSON API? TRY A SCHEMALESS STRUCTURED BINARY FORMAT! IT’S EASY.