Effective akka scalaio

44
@jamie_allen Eective Actors Jamie Allen Director of Consulting

Transcript of Effective akka scalaio

Page 1: Effective akka scalaio

@jamie_allen

Effective Actors

Jamie AllenDirector of Consulting

Page 2: Effective akka scalaio

@jamie_allen 2

Page 3: Effective akka scalaio

@jamie_allen

Goal

Communicate  as  much  about  what  I’ve  learned  in  4+  years  of  actor  development  within  one  

hour

NOTE  -­‐  this  is  not  an  introductory  talk,  and  will  contain  some  advanced  concepts  about  Akka  &  

asynchronous  development

3

Page 4: Effective akka scalaio

@jamie_allen

Extra and Cameo Patterns

A  couple  of  patterns  for  how  to  handle  individual  messages  to  an  actor  without  making  it  do  all  of  the  work  before  handling  the  next  

message

4

Page 5: Effective akka scalaio

@jamie_allen

Request Aggregation•When  an  actor  handles  a  message,  it  frequently  has  to  perform  multiple  interactions  with  other  services  to  provide  a  valid  response

•We  do  not  want  the  actor  that  received  the  message  to  be  tied  up  performing  that  work

•What  are  our  options?

5

Page 6: Effective akka scalaio

@jamie_allen

Use Case•An  actor  will  receive  a  request  to  get  all  of  the  account  balances  for  a  customer  (savings,  checkings  and  money  market)

•Actor  should  not  wait  to  finish  handling  one  request  before  handling  another

•Actor  should  receive  service  proxies  from  which  it  can  retrieve  each  account’s  info

•Actor  should  either  get  responses  from  all  three  within  a  specified  time,  or  send  a  timeout  response  back  to  the  requestor

6

Page 7: Effective akka scalaio

@jamie_allen

Request Aggregation

7

Request Receiver

Message 1 Handler

Message 2 Handler

Message 3 Handler

Message 1

Message 4

Message 2

Message 3

Message 4 Handler

Message 1 Handler

Page 8: Effective akka scalaio

@jamie_allen

Transactions?•Could  be!•You  have  to  write  the  logic  of  how  to  roll  back  if  anything  fails

•But  you  have  the  control  and  the  context  to  do  it,  especially  if  your  effects  are  going  to  multiple  external  places  or  data  stores

8

Page 9: Effective akka scalaio

@jamie_allen

All Code is on GitHub• I’m  going  to  be  showing  some  reasonably  complex  examples• Don’t  worry  about  trying  to  copy  the  code  from  the  slides

http://github.com/jamie-allen/effective_akka

9

Page 10: Effective akka scalaio

@jamie_allen

Use Futures and Promises?•No!    Use  another  actor  responsible  for:

•Capturing  the  context  (original  requestor)•Defining  how  to  handle  responses  from  other  services

•Defining  the  timeout  that  will  race  against  your

•Each  future  and  the  resulting  promise  have  an  additional  cost  that  we  do  not  need  to  pay

10

Page 11: Effective akka scalaio

@jamie_allen

Use Futures and Promises? def receive = { case GetCustomerAccountBalances(id) => val futSavings = savingsAccounts ? GetCustomerAccountBalances(id) val futChecking = checkingAccounts ? GetCustomerAccountBalances(id) val futMM = moneyMarketAccounts ? GetCustomerAccountBalances(id)

val futBalances = for { savings <- futSavings.mapTo[Option[List[(Long, BigDecimal)]]] checking <- futChecking.mapTo[Option[List[(Long, BigDecimal)]]] mm <- futMM.mapTo[Option[List[(Long, BigDecimal)]]] } yield AccountBalances(savings, checking, mm) futBalances map (sender ! _) }}

11

Yuck!

Page 12: Effective akka scalaio

@jamie_allen

Capturing the Sender•This  is  trickier  than  it  sounds•You  need  the  “sender”  value  from  the  actor  that  received  the  original  request,  not  the  sender  inside  of  the  actor  handling  it!

•One  of  the  biggest  sources  of  actor  bugs

12

Page 13: Effective akka scalaio

@jamie_allen

Use Futures and Promises? def receive = { case GetCustomerAccountBalances(id) => val futSavings = savingsAccounts ? GetCustomerAccountBalances(id) val futChecking = checkingAccounts ? GetCustomerAccountBalances(id) val futMM = moneyMarketAccounts ? GetCustomerAccountBalances(id)

val futBalances = for { savings <- futSavings.mapTo[Option[List[(Long, BigDecimal)]]] checking <- futChecking.mapTo[Option[List[(Long, BigDecimal)]]] mm <- futMM.mapTo[Option[List[(Long, BigDecimal)]]] } yield AccountBalances(savings, checking, mm) futBalances map (sender ! _) }

13

Bug!

Page 14: Effective akka scalaio

@jamie_allen

Use an Anonymous Actor?class OuterActor extends Actor def receive = LoggingReceive { case DoWork => { val originalSender = sender

context.actorOf(Props(new Actor() { def receive = LoggingReceive { case HandleResponse(value) => timeoutMessager.cancel sendResponseAndShutdown(Response(value)) case WorkTimeout => sendResponseAndShutdown(WorkTimeout) }

def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) }

someService ! DoWork

import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! WorkTimeout } })) } }}

14

} Anonymous  Actor

Page 15: Effective akka scalaio

@jamie_allen

Use an Anonymous Actor?•I  call  this  the  “Extra”  Pattern•It’s  not  bad,  but  it  has  drawbacks:

•Poor  stack  traces  due  to  “name  mangled”  actor  names,  like  $a

•More  difficult  to  maintain  the  cluttered  code,  and  developers  have  to  read  through  the  body  of  the  anonymous  actor  to  figure  out  what  it  is  doing

•More  likely  to  “close  over”  state

15

Page 16: Effective akka scalaio

@jamie_allen

Anonymous Actor Exampleclass AccountBalanceRetrieverFinal(savingsAccounts: ActorRef, checkingAccounts: ActorRef, moneyMarketAccounts: ActorRef) extends Actor with ActorLogging def receive = LoggingReceive { case GetCustomerAccountBalances(id) => { val originalSender = sender

context.actorOf(Props(new Actor() { var checkingBalances, savingsBalances, mmBalances: Option[List[(Long, BigDecimal)]] = None def receive = LoggingReceive { case CheckingAccountBalances(balances) => checkingBalances = balances collectBalances case SavingsAccountBalances(balances) => savingsBalances = balances collectBalances case MoneyMarketAccountBalances(balances) => mmBalances = balances collectBalances case AccountRetrievalTimeout => sendResponseAndShutdown(AccountRetrievalTimeout) }

def collectBalances = (checkingBalances, savingsBalances, mmBalances) match { case (Some(c), Some(s), Some(m)) => timeoutMessager.cancel sendResponseAndShutdown(AccountBalances(checkingBalances, savingsBalances, mmBalances)) case _ => }

def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) }

savingsAccounts ! GetCustomerAccountBalances(id) checkingAccounts ! GetCustomerAccountBalances(id) moneyMarketAccounts ! GetCustomerAccountBalances(id)

import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! AccountRetrievalTimeout } })) } }

16

Page 17: Effective akka scalaio

@jamie_allen

Create a “Cameo” Actor•Externalize  the  behavior  of  such  an  anonymous  actor  into  a  specific  type

•Anyone  maintaining  the  code  now  has  a  type  name  from  which  they  can  infer  what  the  actor  is  doing

•Most  importantly,  you  can’t  close  over  state  from  an  enclosing  actor  -­‐  it  must  be  passed  explicitly

17

Page 18: Effective akka scalaio

@jamie_allen

Create a “Cameo” Actorclass WorkerActor extends Actor { def receive = LoggingReceive { case HandleResponse(value) => timeoutMessager.cancel sendResponseAndShutdown(Response(value)) case WorkTimeout => sendResponseAndShutdown(WorkTimeout) }

def sendResponseAndShutdown(response: Any) = { originalSender ! response context.stop(self) }

import context.dispatcher val timeoutMessager = context.system.scheduler.scheduleOnce(250 milliseconds) { self ! WorkTimeout }}

class DelegatingActor extends Actor def receive = LoggingReceive { case DoWork => { val originalSender = sender val worker = context.actorOf(WorkerActor.props(), “worker”) someService.tell(DoWork, worker) } }}

18

Page 19: Effective akka scalaio

@jamie_allen

Remember to Stop the Actor•When  you  are  finished  handling  a  request,  ensure  that  the  actor  used  is  shutdown• This  is  a  big  memory  leak  if  you  don’t

def sendResponseAndShutdown(response: Any) = { originalSender ! response log.debug("Stopping context capturing actor") context.stop(self) }

19

Page 20: Effective akka scalaio

@jamie_allen

Write Tests!•Always  remember  to  write  tests  with  your  actors

•Create  unit  tests  that  check  the  functionality  of  method  calls  without  actor  interations  using  TestActorRef

•Create  integration  tests  that  send  messages  to  actors  and  check  the  aggregated  results

20

Page 21: Effective akka scalaio

@jamie_allen

Write Tests!class CameoSpec extends TestKit(ActorSystem("cameo-test-as")) with ImplicitSender with WordSpecLike with MustMatchers { val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "checkings") val moneyMarketAccountsProxy = system.actorOf(Props[MoneyMarketAccountsProxyStub], "money-markets")

"An AccountBalanceRetriever" should { "return a list of account balances" in { val probe1 = TestProbe() val probe2 = TestProbe() val savingsAccountsProxy = system.actorOf(Props[SavingsAccountsProxyStub], "cameo-savings") val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "cameo-checkings") val moneyMarketAccountsProxy = system.actorOf(Props[MoneyMarketAccountsProxyStub], "cameo-mm") val accountBalanceRetriever = system.actorOf(Props(new AccountBalanceRetriever(savingsAccountsProxy, checkingAccountsProxy, moneyMarketAccountsProxy)), "cameo-retriever1")

within(300 milliseconds) { probe1.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) val result = probe1.expectMsgType[AccountBalances] result must equal(AccountBalances(Some(List((3, 15000))), Some(List((1, 150000), (2, 29000))), Some(List()))) } within(300 milliseconds) { probe2.send(accountBalanceRetriever, GetCustomerAccountBalances(2L)) val result = probe2.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((6, 640000), (7, 1125000), (8, 40000))), Some(List((5, 80000))), Some(List((9, 640000), (10, 1125000), (11, 40000))))) } } }}

21

Page 22: Effective akka scalaio

@jamie_allen

Write Tests! within(300 milliseconds) { probe1.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) val result = probe1.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((3, 15000))), Some(List((1, 150000), (2, 29000))), Some(List()))) } within(300 milliseconds) { probe2.send(accountBalanceRetriever, GetCustomerAccountBalances(2L)) val result = probe2.expectMsgType[AccountBalances] result must equal(AccountBalances( Some(List((6, 640000), (7, 1125000), (8, 40000))), Some(List((5, 80000))), Some(List((9, 640000), (10, 1125000), (11, 40000))))) }

22

Page 23: Effective akka scalaio

@jamie_allen

Write Moar Tests!

23

"return a TimeoutException when timeout is exceeded" in { val checkingAccountsProxy = system.actorOf(Props[CheckingAccountsProxyStub], "cameo-timing-out-checkings")

within(250 milliseconds, 500 milliseconds) { probe.send(accountBalanceRetriever, GetCustomerAccountBalances(1L)) probe.expectMsg(AccountRetrievalTimeout) } }

Page 24: Effective akka scalaio

@jamie_allen

Best PracticesSeveral  important  learnings  I’ve  gleaned  in  my  

years  of  actor  development

24

Page 25: Effective akka scalaio

@jamie_allen

Create a Props Factory• Creating  an  actor  within  another  actor  implicitly  closes  over  “this”• Necessary  until  spores  (SIP-­‐21)  are  part  of  Scala• Create  a  Props  factory  in  a  companion  object• Anonymous  actors  can’t  have  one!object AccountBalanceResponseHandler { def props(savingsAccounts: ActorRef, checkingAccounts: ActorRef, moneyMarketAccounts: ActorRef, originalSender: ActorRef): Props = {

Props(new AccountBalanceResponseHandler(savingsAccounts, checkingAccounts, moneyMarketAccounts, originalSender)) }}

25

Page 26: Effective akka scalaio

@jamie_allen

Keep Your Actors Simple• Do  not  conflate  responsibilities  in  actors

• Becomes  hard  to  define  the  boundaries  of  responsibility

• Supervision  becomes  more  difficult  as  you  handle  more  possibilities

• Debugging  becomes  very  difficult

26

Page 27: Effective akka scalaio

@jamie_allen

Be Explicit in Supervision• Every  non-­‐leaf  node  is  technically  a  

supervisor

• Create  explicit  supervisors  under  each  node  for  each  type  of  child  to  be  managed

27

Page 28: Effective akka scalaio

@jamie_allen

Conflated Supervision

28

C2C1

D1 D2 D3Ac2Ac1

A1 A2 A3 A4

Customers

Accounts & Devices

Applications

Page 29: Effective akka scalaio

@jamie_allen

Explicit Supervision

29

C2C1

D1 D2 D3Ac2Ac1

A1 A2 A3 A4

Customers

Accounts & Devices

Applications

AS DS

Page 30: Effective akka scalaio

@jamie_allen

Use Failure Zones• Multiple  isolated  zones  with  their  own  

resources  (thread  pools,  etc)

• Prevents  starvation  of  actors

• Prevents  issues  in  one  branch  from  affecting  another

val responseHandler = system.actorOf( AccountBalanceResponseHandler.props(), "cameo-handler").withDispatcher( "handler-dispatcher")

30

Page 31: Effective akka scalaio

@jamie_allen

No Failure Zones

31

C2C1

D1 D2 D3Ac2Ac1

A1 A2 A3 A4

Customers

Accounts & Devices

Applications

AS DS

Page 32: Effective akka scalaio

@jamie_allen

Explicit Failure Zones

32

C2C1

D1 D2 D3Ac2Ac1

A1 A2 A3 A4

Customers

Applications

AS DS

Accounts & Devices

Page 33: Effective akka scalaio

@jamie_allen

Push, Don’t Pull• Start  with  no  guarantees  about  delivery

• Add  guarantees  only  where  you  need  them

• Retry  until  you  get  the  answer  you  expect

• Switch  your  actor  to  a  "nominal"  state  at  that  point

33

Page 34: Effective akka scalaio

@jamie_allen

Beware the Thundering Herd• Actor  systems  can  be  overwhelmed  by  "storms"  of  

messages  flying  about

• Do  not  pass  generic  messages  that  apply  to  many  actors

• Dampen  actor  messages  if  the  exact  same  message  is  being  handled  repeatedly  within  a  certain  timeframe

• Tune  your  dispatchers  and  mailboxes  via  back-­‐off  policies  and  queue  sizes

• Akka  has  Circuit  Breakers  to  help  you  provide  back-­‐pressure

34

Page 35: Effective akka scalaio

@jamie_allen

Create Granular Messages

35

• Non-­‐specific  messages  about  general  events  are  dangerous

AccountDeviceAdded(acctNum, deviceNum)

AccountsUpdated

• Can  result  in  "event  storms"  as  all  actors  react  to  them

• Use  specific  messages  forwarded  to  actors  for  handling

Page 36: Effective akka scalaio

@jamie_allen

Unique IDs for Messages• Allows  you  to  track  message  flow

• When  you  find  a  problem,  get  the  ID  of  the  message  that  led  to  it

• Use  the  ID  to  grep  your  logs  and  display  output  just  for  that  message  flow

• Akka  ensures  ordering  on  a  per  actor  basis,  also  in  logging

36

Page 37: Effective akka scalaio

@jamie_allen

Create Specialized Exceptions• Don't  use  java.lang.Exception  to  represent  

failure  in  an  actor

• Specific  exceptions  can  be  handled  explicitly

• State  can  be  transferred  between  actor  incarnations  in  Akka  (if  need  be)

37

Page 38: Effective akka scalaio

@jamie_allen

Never Reference “this”• Actors  die

• Doesn't  prevent  someone  from  calling  into  an  actor  with  another  thread

• Akka  solves  this  with  the  ActorRef  abstraction

• Never  expose/publish  “this”

• Loop  by  sending  messages  to  “self”

• Register  by  sending  references  to  your  “self”

38

Page 39: Effective akka scalaio

@jamie_allen

Immutable Interactions•All  messages  passed  between  actors  should  be  immutable

•All  mutable  data  to  be  passed  with  messages  should  be  copied  before  sending

•You  don’t  want  to  accidentally  expose  the  very  state  you’re  trying  to  protect

39

Page 40: Effective akka scalaio

@jamie_allen

Externalize Logic• Consider  using  external  functions  to  

encapsulate  complex  business  logic

• Easier  to  unit  test  outside  of  actor  context

• Not  a  rule  of  thumb,  but  something  to  consider  as  complexity  increases

• Not  as  big  of  an  issue  with  Akka's  TestKit

40

Page 41: Effective akka scalaio

@jamie_allen

Semantically Useful Logging• Trace-­‐level  logs  should  have  output  that  you  

can  read  easily

• Use  line  breaks  and  indentation

• Both  Akka  and  Erlang  support  hooking  in  multiple  listeners  to  the  event  log  stream

41

Page 42: Effective akka scalaio

@jamie_allen

Use the Typesafe Console in Development

•Visual  representations  of  actors  at  runtime  are  invaluable  tools

•Keep  an  eye  out  for  actors  whose  mailboxes  never  drain  and  keep  getting  bigger

•Look  out  for  message  handling  latency  that  keeps  going  higher

•These  are  signs  that  the  actors  cannot  handle  their  load•Optimize  with  routers•Rethink  usage  of  Dispatchers

42

Page 43: Effective akka scalaio

@jamie_allen

Use the Typesafe Console in Development

43

Page 44: Effective akka scalaio

@jamie_allen

Thank You!• Some  content  provided  by  members  of  the  

Typesafe  team,  including:

• Jonas  Bonér

• Viktor  Klang

• Roland  Kuhn

• Havoc  Pennington

44