Building MapAttack: A real-time geolocation

gameKyle Drake

Sunday, February 12, 12

geoloqi.com

What is Geoloqi?

h!p://geoloqi.com

Sunday, February 12, 12

geoloqi.com

Real-world solutions for thousands around the globe

Sunday, February 12, 12

geoloqi.com

Real-World Interactive Experiences

Real-Time Location Tracking

Sunday, February 12, 12

geoloqi.com

Personal Location

Sunday, February 12, 12

geoloqi.com

Location-Based Notes

Sunday, February 12, 12

geoloqi.com

YOUR GPS GETS WEIRD.

WE HELP FIX THAT.

Sunday, February 12, 12

geoloqi.com

Real-Time Applications

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

First Hackathon Attempt

Layer in Geoloqi

Updates to MapAttack server via HTTP REST API

Processing through Geoloqi’s persistent store

Finished in a weekend

A lot of fun!

Sunday, February 12, 12

Sunday, February 12, 12

Sunday, February 12, 12

Sunday, February 12, 12

Sunday, February 12, 12

geoloqi.com

There were some kinks Used an REST HTTP service for

APNS (worked, but not “real-time”)

Server was getting backed up (updates were not async)

Sunday, February 12, 12

geoloqi.com

MapAttack Server Issues

Blocking IO (requests to Geoloqi API held everything up)

Lots of concurrent traffic

Events had to be broadcast to all phones/clients in the game synchronously

Sunday, February 12, 12

geoloqi.com

HOW WE DEALT WITH

ITSunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

A key-value store, and so much more

A great implementation of PUBLISH/SUBSCRIBE

So simple, you can actually understand it

Very fast and reliable

Robust library support

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

• Phones:Custom binary protocol over UDP

• Web Browsers:Web Sockets via Socket.IO

HOW WE USE NODE JS

Sunday, February 12, 12

geoloqi.com

Socket.IO is awesome Adapter Pattern for realtime

Web Sockets, Flash, Long Polling!

One interface for everything

Easy to implement: ~100 LOC

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

Geoloqi API ASYNC FTW!

Delayed updates when possible

Messages queues: Beanstalk

Workers pick tasks off the stack

Process data outside of the persistent store

Simpler than AMQP

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

The Reactor Pattern“The reactor design pattern is a concurrent programming pattern for handling service

requests delivered concurrently to a service handler by one or more inputs” - Wikipedia

My simplest description: Takes your blocking IO operation, shoves it into its own kernel thread

behind the scenes, uses Unix kernel magic to make it rejoin the reactor queue when it’s ready.

Linux: epoll(4)

BSD: kqueue/keventSunday, February 12, 12

geoloqi.com

Concurrent IO is a common problem

All programming languages have trouble with it

The Reactor pattern is a tool to resolve it

Most languages have the Reactor pattern!

JavaScript Node.js

Python Twisted

Ruby EventMachine (libem, C)

Java JBoss_Netty

PHP None yet

Sunday, February 12, 12

geoloqi.com

MapAttack Server

MRI Ruby has a global interpreter lock

But MRI Ruby does not block on IO

JRuby and Rubinius 2: Native Threads

And it has a reactor pattern, just like Node JS

(or: How to make Ruby ROFLscale)

Sunday, February 12, 12

geoloqi.com

Sinatra::Synchrony

Implements EventMachine with Fibers

The result: NO CALLBACKS

< 100 LOC

Only coding change is to use EM libs and fibers

DON’T USE IT.

kyledrake.net/sinatra-synchrony

Sunday, February 12, 12

geoloqi.com

Sinatra::Synchrony

Sunday, February 12, 12

geoloqi.com

Sunday, February 12, 12

geoloqi.com

Let’s Talk Performance.

Sunday, February 12, 12

geoloqi.com

Database = your real persistence problem

Sunday, February 12, 12

geoloqi.com

Make the relevant data stay outside of the slow

persistence store.

Sunday, February 12, 12

geoloqi.com

If you’re trying to solve a problem by ROFLscaling your persistent store,

you’re doing it wrong.

Sunday, February 12, 12

geoloqi.com

Examples of doing it wrong “Let’s rewrite everything to use TrendDB”

Using EC2 for a single-master DB “because it scales”

Buying something from Oracle

“Ruby/PHP/JS is too slow for this, let’s rewrite it in TrendLang”

Many NoSQL solutions have problems you don’t know about. Global write locks, single write master, et cetera

Sunday, February 12, 12

geoloqi.com

If you must speed up your persistent store, here’s my practical advice.

Sunday, February 12, 12

geoloqi.com

Real Hardware FTW? Single master database? Get a real server! Really!

Cloud is only as fast as the fastest available HDD

EBS performance is not great, even in RAID0 stripeSee orion.heroku.com/past/2009/7/29/io_performance_on_ebsand perfcap.blogspot.com/2011/03/understanding-and-using-amazon-ebs.htmland blog.dt.org/index.php/2010/06/amazon-ec2-io-performance-local-emphemeral-disks-vs-raid0-striped-ebs-volumes/and endevver.com/2010/03/cost-analysis-of-an-amazon-ec2-deployment.htmland mysqlperformanceblog.com/2011/02/21/death-match-ebs-versus-ssd-price-performance-and-qosand google.com/search?q=heroku+ebs+performance

High end Xeon, 32GB ECC, fast 220GB SSD: ~$2K

Sunday, February 12, 12

geoloqi.com

Real Hardware FTW?The paradigm is diversifying, and the reason is SSD.

Say hello to the Fusion-io ioDrive Octal.PCIe x16 slot5.12TB6.0 GB/s read4.4 GB/s write1.19 MILLION IOPS

Cost: $100,000 (plus $10 S&H).It’s a steal at this price.. but it will get cheaper.

Sunday, February 12, 12

geoloqi.com

Real Hardware FTW?$500 PCIe SSD drives:

220GB>100,000 IOPS~700MB/s R/W

15,000 RPM SAS drives:2TB+

~175-210 IOPS<150MB/s R/W

Sunday, February 12, 12

geoloqi.com

Long Term Multi-master experiment (Brewer’s CAP)

Riak is my current favorite

Cloud companies providing real hardware performance

Map/Reduce != Database

Sunday, February 12, 12

geoloqi.com

In Summary

Sunday, February 12, 12

geoloqi.com

Geo-location gamesare FUN!

Sunday, February 12, 12

geoloqi.com

Use the Geoloqi API to make geolocation games! We did the

hard work for you.

Sunday, February 12, 12

geoloqi.com

MapAttack is open source!github.com/geoloqi/MapAttack

Sunday, February 12, 12

geoloqi.com

Other uses of Geoloqi

Sunday, February 12, 12

geoloqi.com

Thanks!

Sunday, February 12, 12