Igloos and Mountains.key

Taking AKKA to ProductionIGL S AND MOUNT NS

Hello! I’m Derek Wyatt, Senior Platform Developer at Auvik Networks!and author of Akka Concurrency

Scala, Play and Akka form the foundational triad for Auvik Networks’ core technology. In this talk, we’ll discuss how Akka enables the asynchronous design of the system and how it powers the production environment.

THIS TALK

However, it’s not all cherries and moonbeams - we faced a number of challenges, including technological hurdles and more than a few spiky PEBKACs. You’ll hear how we overcame these issues, as well as provide some insight into how we’re growing our understanding and practices around the Typesafe platform.

Auvik Networks

Auvik simplifies monitoring, configuring and automating small and medium sized enterprise networks. We help IT professionals build Networks That Work.TM

AUVIK AND THE TYPESAFE STACK

Auvik Networks

Auvik simplifies monitoring, configuring and automating small and medium sized enterprise networks. We help IT professionals build Networks That Work.TM

Auvik is all in with Akka, Scala and Play - our future success is tied to the success of the Typesafe Platform. Akka’s design is what powers Auvik’s design.

AUVIK AND THE TYPESAFE STACK

Hybrid Cloud SaaS

Auvik’s solution is half in the cloud and half on site

30K FOOT ARCHITECTURE

Hybrid Cloud SaaS


The cloud provides the bulk of the operational logic and the on site software works on behalf of the cloud


Hybrid Cloud SaaS



Technically, the on site software can operate disconnected from the cloud, but eventually it has to connect in, otherwise it’s working for nothing


Hybrid Cloud SaaS



Technically, the on site software can operate disconnected from the cloud, but eventually it has to connect in, otherwise it’s working for nothing

While the on site component is an Akka solution, it does not communicate with the cloud using Akka protocols, since these aren’t design for the WAN


Auvik’s core design is an Actor-based design

Customers exist in distinct Actor subtrees.

Tables, On Site connections, the Execution Engine, and many other components are all Actors

AUVIK USES AKKA




Those Actors ensure that our customer design is…

AUVIK USES AKKA





AUVIK USES AKKA

Isolated





AUVIK USES AKKA

Resilient

Isolated





AUVIK USES AKKA

Asynchronous

Resilient

Isolated





AUVIK USES AKKA

Asynchronous

ResilientNon Blocking

Isolated





AUVIK USES AKKA

Asynchronous

ResilientNon Blocking

IsolatedReasonable

The Cluster

CLOUD DEPLOYMENT

The Cluster

We use Akka Clustering to manage our node fabric

CLOUD DEPLOYMENT

The Cluster


Everything runs in Amazon, defined by CloudFormation and configured by Chef

CLOUD DEPLOYMENT

The Cluster



Akka Clustering doesn’t survive across a WAN, so communication with the onsite component is proprietary

CLOUD DEPLOYMENT

The Cluster




Backends and Frontends are specified to scale independently

CLOUD DEPLOYMENT

The Cluster




Backends and Frontends are specified to scale independently

Cluster discovery happens through a well-known seed node (not shown)

CLOUD DEPLOYMENT

Cluster Singletons Customers would be great Cluster Singletons, except Cluster Singletons don’t load balance

CHALLENGES AND SURPRISES


Inner classes and closures Inner classes and closures are awesome, unless they get caught up in messages and force us to serialize the planet


Application Debugging asynchronous applications is still hard. Bulkhead your application to protect it from itself





Performance issues Tracking down performance issues can be tough - that ForkJoin pool may not be your friend




Play Framework Play doesn’t play nicely with Actor applications






Play Framework Play doesn’t play nicely with Actor applications

Future Correctness & Await.result Await.result is just evil. It’s not “sorta” evil… it’s the devil with the flu.






Our own Cluster Singleton

MANAGING CLUSTER CUSTOMERS


We created our own way of managing customers as “single instances”




The guy that manages the customers is a proper Akka Cluster Singleton





He manages the instantiation of the customers on-demand





He manages the instantiation of the customers on-demand

He also reacts to Akka Cluster events (new nodes, dead nodes, etc…) and can rebalance customers as required


Scala makes this just too damn easy…

CLOSING OVER THE WORLD…

trait MyApp { case class Question(s: String) case class Answer(s: String) ! class Server extends Actor { def receive = { case Question(q) => sender ! Answer(s”Sup? $q”) } } class Client(server: ActorRef) extends Actor { server ! Question(“Dood!”) def receive = { case Answer(a) => println(s”Got $a”) } } }




Chances are that Question and

Answer won’t serialize




Chances are that Question and

Answer won’t serializeAnd even if they did,

they’d be huge

PLAY DOESN’T PLAY NICELY

Modular Design


Modular DesignPlay doesn’t start like “just any other app”

Special developer modeSpecial run modeEither way, it’s “special”




We want to start like a “normal” appBut not for developersFrontend needs to start backed when the Play Plugin starts

But not when it’s running in Frontend standalone mode






Bootstrapping the configuration differs depending on how we start it






Bootstrapping the configuration differs depending on how we start it

Modes for SBT

Main app for Production

IDE Configurations

DEBUGGING IS HARD

Captain Obvious…Akka makes writing concurrent software easier and clearer, but it doesn’t necessarily make it easier to debug

DEBUGGING IS HARD


We had tons of work being done in the ForkJoin scheduler, but why?

DEBUGGING IS HARD

The overhead of the Future had better be worth it!



DEBUGGING IS HARD




We had a lot of timeouts occurring, but why?

DEBUGGING IS HARD


A chain of N futures can time out, but knowing which one timed out isn’t obvious




DEBUGGING IS HARD






We had race conditions that “shouldn’t be there”, but why?

DEBUGGING IS HARD



Message ordering guarantees are only between two Actors (adding a Future in the middle can warp your perception)





DEBUGGING IS HARD








We had Exceptions being thrown in messaging processors but messages sent by what?

DEBUGGING IS HARD




Dropping a crappy message in a mailbox is perfectly correct; processing it is disconnected with queueing it





We had Exceptions being thrown in messaging processors but messages sent by what?

DEBUGGING IS HARD

Side effecting Futures

DEBUGGING IS HARD

Side effecting FuturesIf you’re plumbing Futures to Actors, avoid side-effects doSomeDBThing() onSuccess {

case result => someActor ! result }

Horrendous!

DEBUGGING IS HARD

Side effecting FuturesIf you’re plumbing Futures to Actors, avoid side-effects

Errors are lost, and onSuccess has no return value

doSomeDBThing() onSuccess { case result => someActor ! result }

Horrendous!

DEBUGGING IS HARD



Always treat Futures as message transformers. Make a clear application protocol and stick with it


doSomeDBThing() map { result => DBResult(result) } recover { case t: Throwable => DBCallFailed(t) } pipeTo someActor

Horrendous!

Absolutely Lovely

DEBUGGING IS HARD



Always treat Futures as message transformers. Make a clear application protocol and stick with it

This is easy to understand, catches the errors, and composes as well as asynchronous code can


doSomeDBThing() map { result => DBResult(result) } recover { case t: Throwable => DBCallFailed(t) } pipeTo someActor

Horrendous!

Absolutely Lovely

DEBUGGING IS HARD

Broken Promises

DEBUGGING IS HARD

Broken Promises“Most” of us shouldn’t be creating a promise

DEBUGGING IS HARD


If you do, make damn sure you complete that promise

val p = Promise[String]() doSomeDBThing() foreach { result => val s = result.get(“someStringValue”) p.success(s) }

Horrendous!

DEBUGGING IS HARD



Using the side effect operations on Futures (for example) is a pretty bad way to do it


Horrendous!

DEBUGGING IS HARD




If you want to plumb a Future to a Promise, this is a much better approach


Horrendous!

Catches Everything

val p = Promise[String]() p.completeWith(doSomeDBThing() map { _.get(“someStringValue”) })

DEBUGGING IS HARD




If you want to plumb a Future to a Promise, this is a much better approach

If you’re not plumbing a Future to a Promise then you’re on your own to make sure it completes… don’t mess it up


Horrendous!

Catches Everything

val p = Promise[String]() p.completeWith(doSomeDBThing() map { _.get(“someStringValue”) })

DEBUGGING IS HARD

The Lost Sender

DEBUGGING IS HARD

The Lost SenderThink of the sender as the recipient of your response, not the subscriber to your stream

DEBUGGING IS HARD


def receive = { case Subscribe() => addSubscriber(sender) sender ! Subscribed() }

DEBUGGING IS HARD


So long as you always call it this way, you’re probably OK (although no guarantees)


publisher ! Subscribe() def receive = { case Subscribed() => println(“Keen!”) }

OK

DEBUGGING IS HARD



But what if you ask? You’re going to subscribe the temporary ask’s Actor!


publisher ! Subscribe() def receive = { case Subscribed() => println(“Keen!”) }

publisher.ask(Subscribe()) .mapTo[Subscribed] .foreach(println)

OK

#Fail

DEBUGGING IS HARD



But what if you ask? You’re going to subscribe the temporary ask’s Actor!

The sender is just too flexible to treat this way, so make it part of your messaging protocol and you’re good to go

OK

def receive = { case Subscribe(subscriber) => addSubscriber(subscriber) sender ! Subscribed(subscriber) }

publisher ! Subscribe(self) def receive = { case Subscribed(`self`) => println(“Keen!”) }

publisher.ask(Subscribe(guy)) .mapTo[Subscribed] .foreach(println)

OK

LESS THAN AWESOME TOOLING

Tools aren’t quite there


Tools aren’t quite thereWe used JVisualVM 10% Effective


Tools aren’t quite thereWe used JVisualVM

We used YourKit

10% Effective

30% Effective



We used YourKit

We built our own tools

Our tools help us debug the proprietary programming language and execution that we’ve created

10% Effective

30% Effective

50% Effective



We used YourKit



That helps us understand how our “app” is working, but it’s also been quite effective to understand how Akka / Java is working

10% Effective

30% Effective

50% Effective



We used YourKit




Bottom line… we’d still like better tools! :)

10% Effective

30% Effective

50% Effective



We used YourKit





10% Effective

30% Effective

50% Effective

Where did this Future come from?



We used YourKit





10% Effective

30% Effective

50% Effective

Where did this Future come from?What Futures have yet to complete?



We used YourKit





10% Effective

30% Effective

50% Effective

Where did this Future come from?What Futures have yet to complete?Is this Actor busy? Blocked?



We used YourKit





10% Effective

30% Effective

50% Effective

Where did this Future come from?What Futures have yet to complete?Is this Actor busy? Blocked?Which Future timed out?



We used YourKit





10% Effective

30% Effective

50% Effective

Where did this Future come from?What Futures have yet to complete?Is this Actor busy? Blocked?Which Future timed out?Where did this message come from?

BULKHEAD!

One Thread Pool never works…

BULKHEAD!


When we start, we code quickly, and JDBC calls go on the same thread pool as everything else

BULKHEAD!



Your blocking calls start chewing up threads that Actors need

BULKHEAD!




So you increase the number of threads, and if they’re ForkJoin threads, well… we’ll get to that

BULKHEAD!





I think we did it right. Go fast, figure things out, then pull the foundations together once you know what you need done

BULKHEAD!





I think we did it right. Go fast, figure things out, then pull the foundations together once you know what you need done

The ExecutionContext is your friend!

Managing Asynchrony

THREADS, FUTURES AND FORKJOIN

Managing Asynchrony


Got Futures?

Managing Asynchrony


Got Futures? Got (lotsa) Threads?

Managing Asynchrony


Got Futures? Got (lotsa) Threads? Got ForkJoin?

Managing Asynchrony



! ! ! Haz Problemz!

Managing AsynchronyThe ForkJoin Pool’s scheduler is O(n) in the number of threads



! ! ! Haz Problemz!


If you need lots of threads, for whatever reason, then scheduling work on the FJ Pool becomes a non-trivial part of your computation



! ! ! Haz Problemz!



Do yourself a favour and configure a standard thread pool executor



! ! ! Haz Problemz!



Do yourself a favour and configure a standard thread pool executor

Our throughput doubled when we did that



! ! ! Haz Problemz!

AWAIT AND DIE

Await != synchronized

AWAIT AND DIE

Await != synchronizedSome people seem to think that an Await.result (on an Actor response) is like a synchronized block

AWAIT AND DIE


Some people are silly :)

AWAIT AND DIE



If you’re waiting for the response to “Your Message”, how long are you gonna wait?

AWAIT AND DIE




You’re waiting behind all the other messages in that mailbox, plus fighting for time from all the other Actors in the system

AWAIT AND DIE





AND you’re chewing up a much-needed thread during that time!

AWAIT AND DIE





AND you’re chewing up a much-needed thread during that time!

Auvik Bear Judges you!

OUR RULES OF THUMB

OUR RULES OF THUMB

Avoid nested classes to help serialization

OUR RULES OF THUMB


Never use Await… well, you can in tests

OUR RULES OF THUMB



Keep Futures as pure as possible; no side-effects

OUR RULES OF THUMB




Bulkhead your IO and other nasty code

OUR RULES OF THUMB





Choose your thread pool types carefully (ForkJoin?)

OUR RULES OF THUMB






sender is the message recipient, not the subscriber to a stream

OUR RULES OF THUMB







Keep things simple and just “let it crash”

OUR RULES OF THUMB








Ask, “How does that work when we’re clustered?”

OUR RULES OF THUMB








Ask, “How does that work when we’re clustered?”

Instrument your code and measure, measure, Measure!

WE’RE HAPPY!

Akka powers our worldOur distributed, asynchronous design is an Akka design

WE’RE HAPPY!


Clustering and scaling out isn’t a simple “side effect” of coding in Akka, but it’s heavily simplified

WE’RE HAPPY!



Message-based network-oriented programming is the killer paradigm for the cloud

WE’RE HAPPY!




Akka has allowed us to program defensively by simply adopting the “let it crash” philosophy

WE’RE HAPPY!




Akka has allowed us to program defensively by simply adopting the “let it crash” philosophy

Basically… we’re quite happy :)

Networks That Work.™

Igloos and Mountains.key

Documents

Transcript of Igloos and Mountains.key