Scala at foursquare

15
Jorge Ortiz @jorgeortiz85 Scala @foursquare

Transcript of Scala at foursquare

Jorge Ortiz@jorgeortiz85

Scala @foursquare

foursquare runs on...

• Scala 2.8.0

• Lift 2.1

• MongoDB

• Twitter open source projects

• ~46k lines of Scala

foursquare server team

• 9 out of 11 had no major prior Scala experience

• Previously: Java, Python, C#

• Most exposed to: ML, Lisp

foursquare loves Scala

• Pimped: Option, List, Map

• Type-safe equality: =?

• BackgroundOperation { ... }

• Throttle(“secret feature!”) { ... }

• Any Java library

2.7.7 => 2.8.0(painless migration)

• Compile: 2.7.7 with -deprecation

• Branch

• Compile 2.8.0 without -deprecation

• Compile 2.8.0 with -Xmigration

• Compile 2.8.0 with -Xwarninit

• Switch

foursquare on Lift

• REST API

• Out-of-the-box:

• Rich Ajax/Comet support

• ORMs for PostgresQL, MongoDB

• Random: Email, Facebook, JSON, utils...

Lift Deployment

• nginx

• HAProxy

• API machines - stateless

• WWW machines - stateful

GC issues

• Stateless API machines

• Tens of millions of requests per day

• Only a dozen or so servers

• 12 GB heap => 15sec GCs *ouch*

GC issues

• GC tuning?

• “There's no way to get around the fact that we're allocating faster than we can collect.”

• Allocating ~200 MB/s

• Opportunity to reduce wasteful allocations

• Better load-balancing strategy

• Cap # of concurrent connections

GC issues

• Stateful WWW machines

• Stateful Lift + Scala closures + Java GC

• Lots of variables getting caught in Session

• val venues: RequestCache[List[Venue]] =

RequestCache(Venue.findAll(...))

MongoDB in Scala

• Some support built-in to Lift

• No expressive query language, so we wrote one:

Venue where (_.mayor eqs userid) modify (_.tags push “I’m mayor!”) updateMulti

Twitter open source

• Kestrel

• Ostrich

• GrabbyHands

• (probably more soon...)

We’re hiringhttp://foursquare.com/jobs

Questions?