Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... ·...
Transcript of Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... ·...
![Page 1: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/1.jpg)
Chrome to Chrome PouchDB
Randall Leeds“tilgovi”
![Page 2: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/2.jpg)
Overview
● Background
● Motivation
● Approach
● Future
![Page 3: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/3.jpg)
Overview
● Background
● Motivation
● Approach
● Future
![Page 4: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/4.jpg)
The Peer-to-Peer Web
![Page 5: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/5.jpg)
The Peer-to-Peer What Now!?
“A peer-to-peer (abbreviated to P2P) computer network is one in which each computer in the network can act as a client or server for the other computers in the network”
– https://en.wikipedia.org/wiki/Peer-to-peer
![Page 6: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/6.jpg)
The (Early) Peer-to-Peer Web
● Early Web was peer-to-peer (kinda)– Universities own and manage Web content
– University researchers access and publish content
– Mainframe is server is client
![Page 7: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/7.jpg)
What the %$*$ happened?
![Page 8: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/8.jpg)
“The Cloud” happened...
![Page 9: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/9.jpg)
Why clouds?
● Server software got complicated
● Server software got expensive
● The edge of the network is flakey
● Collective Intelligence, Big Data, blah blah... aggregate all the things... whatever...
![Page 10: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/10.jpg)
The Cloud is a Hack
![Page 11: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/11.jpg)
The Cloud is a Hack
● Connectivity– We're too far from one another!
● Scale– We want to aggregate our data!
● Management– We don't want to do any work!
![Page 12: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/12.jpg)
Fog Computing
“... and Its Role in the Internet of Things”Bonomi, Milito, Zhu, Addepalli
Cisco Systems, Inc.2013
![Page 13: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/13.jpg)
Fog Computing
● Low latency and location awareness
● Wide-spread geographical distribution
● Mobility
● Very large number of nodes
● Predominant role of wireless access
● Strong presence of streaming and real time applications
● Heterogeneity
![Page 14: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/14.jpg)
Overview
● Background
● Motivation
● Approach
● Future
![Page 15: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/15.jpg)
Where did I come from?
![Page 16: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/16.jpg)
Where did I come from?
![Page 17: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/17.jpg)
Where did I come from?
● CouchDB Cluster/Partition GsoC– Proposal no longer available online...
– … which is a good thing.
● Meebo / CouchDB-Lounge– No longer developed...
– … which is a good thing.
![Page 18: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/18.jpg)
Hey, wait!
That's all “cloud stuff”!
![Page 19: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/19.jpg)
The “Indie Web”
(You've probably never heard of it...)
![Page 20: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/20.jpg)
The “Indie Web”Own your data.
Rather than posting content on third-party silos of content, we should all own the content we're
creating. Publish on your own domain, and syndicate out to silos.
This is the basis of the "Indie Web" movement.
Source: http://indiewebcamp.com/
![Page 21: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/21.jpg)
It's hard being cool
![Page 22: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/22.jpg)
It's hard being cool
● The Indie Web has fallen behind
● Running a server is hard.– Like, really, really hard.
– No, really.
![Page 23: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/23.jpg)
Example: Diaspora
Preparing your system
In order to run Diaspora, you will need to install the following dependencies (specific instructions follow):
● Build tools - Packages needed to compile the components that follow.
● Ruby - The Ruby programming language. ( 1.9.2 or later).
● RubyGems - A package manager for Ruby code that we use to download libraries ("gems") that Diaspora uses.
● Bundler - A gem management tool for Ruby projects.
● MySQL - Backend storage engine.
● Or: PostgreSQL - Backend storage engine.
● SQLite3 - Relational database management system
● OpenSSL - An encryption library.
● libcurl - A library to make HTTP requests (and much more).
● ImageMagick - An image processing library we use to resize uploaded photos.
● Git - A version control system, which you will need to download the Diaspora source code from GitHub.
● Redis - A persistent key-value store that we use via Resque for background job processing.
![Page 24: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/24.jpg)
We Can Do (Much) Better
CouchApp (Persistence + Application Server)PouchDB (CouchDB in the browser)
![Page 25: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/25.jpg)
Introducing: PouchDB Server
The Couch is the Browser is the CouchThis ultimate “unhosting”
![Page 26: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/26.jpg)
PouchDB Server
![Page 27: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/27.jpg)
Overview
● Background
● Motivation
● Approach
● Future
![Page 28: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/28.jpg)
PouchDB Server
● Chrome 24 (currently dev) Packaged Apps– chrome.experimental.socket.*
– chrome.experimental.udp.*
● PouchDB
![Page 29: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/29.jpg)
How is this possible?
Mostly, it's kinda easy.
![Page 30: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/30.jpg)
(For Some Value of Easy)
![Page 31: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/31.jpg)
Protip: Steal everything
● Don't reinvent the wheel
● Use what people love
● There are no tests like the tests someone else wrote for you
![Page 32: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/32.jpg)
![Page 33: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/33.jpg)
Moar Javascript
● Node.JS has a stable HTTP and Socket API
● Node.JS is JavaScript
![Page 34: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/34.jpg)
Can we put Node.JSin the browser???
![Page 35: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/35.jpg)
![Page 36: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/36.jpg)
What can we reuse?
Node.JS is largely written in JavaScript!
![Page 37: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/37.jpg)
What can we reuse?
net.js and http.js
![Page 38: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/38.jpg)
Enter Browserify
![Page 39: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/39.jpg)
Browserify
● Write code like you are using node modules
● Run a server-side build step
● Get a bundled JavaScript file
![Page 40: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/40.jpg)
<Jazz Hands>
![Page 41: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/41.jpg)
Problem
Doesn't include 'net' or 'http'
![Page 42: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/42.jpg)
Introducing: chromify
● Browserify plugin adds node's 'net' and 'http' modules
● Unmodified code from node.js core!
● Fake C bindings!
● But... we need an HTTP parser!
![Page 43: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/43.jpg)
Enter Emscripten
● LLVM JS compiler→– (Really, how cool is that?)
● joyent/http-parser– No dependencies
– No system calls
![Page 44: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/44.jpg)
Game Plan
● Step 1:– Monkey patch process.binding()
● Fake 'tcp_wrap', 'cares_wrap' in JS● Make 'http_parser' wrapper around emscripten parser
● Step 2: <hella jazz hands>
● Step 3: profit!
![Page 45: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/45.jpg)
![Page 46: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/46.jpg)
Overview
● Background
● Motivation
● Approach
● Future
![Page 47: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/47.jpg)
What's next?
● Fix all the super-jank
● More exposure– TCP over UDP/STUN
● Overlays
– Proxy gateways
● Other browsers● PouchApps● Parties. All the time.
![Page 48: Randall Leeds “tilgovi”conf.couchdb.org/couchdb-conf-berlin-january-2013/slides/... · 2013-12-04 · CouchDB Cluster/Partition GsoC ... Rather than posting content on third-party](https://reader034.fdocuments.us/reader034/viewer/2022050422/5f91ba34593f462c053f0e5c/html5/thumbnails/48.jpg)
Thank Youhttps://github.com/tilgovi/chromifyhttps://github.com/tilgovi/pouchdb
(pouch-server branch)https://github.com/tilgovi/http-parser
(es-parser branch)