Akka streams scala italy2015

58
Mirco Dotta @mircodotta Akka Streams Asynchronous non-blocking streaming made easy 1

Transcript of Akka streams scala italy2015

Page 1: Akka streams scala italy2015

Mirco Dotta @mircodotta

Akka StreamsAsynchronous non-blocking streaming made easy

1

Page 2: Akka streams scala italy2015

Reactive  Applications

The Four Reactive Traits

http://reactivemanifesto.org/

2

Page 3: Akka streams scala italy2015

Why Reactive?

3

Page 4: Akka streams scala italy2015

Why Reactive?

• Users expectations have changed • Services must be always up. • Must be fast.

• Billions of internet connected devices. • Data is transformed and pushed continuously.

4

Page 5: Akka streams scala italy2015

Reactive Streams

An initiative for providing Standardised(!) Back-pressured Asynchronous

Stream processing !

http://www.reactive-streams.org/

5

Page 6: Akka streams scala italy2015

Reactive Streams: Who?!

• Kaazing • Netflix (rxJava) • Pivotal (reactor) • RedHat (vert.x) • Twitter • Typesafe (akka-streams & slick)

• Play 2.4 also supports reactive streams! • Doug Lea proposed an implementation for JDK9! !

!

6

Page 7: Akka streams scala italy2015

Standardised!

7

Page 8: Akka streams scala italy2015

Reactive Streams: Inter-op!

We want to make different implementations co-operate with each other.

8

Page 9: Akka streams scala italy2015

Reactive Streams: Inter-op

The different implementations “talk to each other” using the Reactive Streams protocol.

9

Page 10: Akka streams scala italy2015

Reactive Streams: Inter-op// here are a few imports that you are not seeing!object ScalaMain extends App {! EmbeddedApp.fromHandler(new Handler {! override def handle(ctx: Context): Unit = {! // RxJava Observable! val intObs = Observable.from((1 to 10).asJava)!! // Reactive Streams Publisher! val intPub = RxReactiveStreams.toPublisher(intObs)!! // Akka Streams Source! val stringSource = Source(intPub).map(_.toString)!! // Reactive Streams Publisher! val stringPub = stringSource.runWith(Sink.fanoutPublisher(1, 1))!! // Reactor Stream! val linesStream = Streams.create(stringPub).map[String](new reactor.function.Function[String, String] {! override def apply(in: String) = in + "\n"! })!! // and now render the HTTP response (RatPack)! ctx.render(ResponseChunks.stringChunks(linesStream))! }!! }).test(new Consumer[TestHttpClient] {! override def accept(client: TestHttpClient): Unit = {! val text = client.getText()! println(text)! system.shutdown()! }! })!} https://github.com/rkuhn/ReactiveStreamsInterop

10

Page 11: Akka streams scala italy2015

Reactive Streams: Inter-op

The Reactive Streams SPI is NOT meant to be user-api. You should use one of the implementing

libraries.

11

Page 12: Akka streams scala italy2015

Akka Streams

12

Page 13: Akka streams scala italy2015

Akka Streams: Basics

• DSL for the formulation of transformations on data streams. • Basic building blocks: • Source  -­‐  something with exactly one output stream. • Flow - something with exactly one input and one output

stream. • Sink - something with exactly one input stream.

• RunnableFlow - A Flow that has both ends “attached” to a Source and Sink respectively, and is ready to be run() .

13

Page 14: Akka streams scala italy2015

Akka Streams: Basics

14

Page 15: Akka streams scala italy2015

Akka Streams: Basics

15

Page 16: Akka streams scala italy2015

Akka Streams: Basics

16

Page 17: Akka streams scala italy2015

Demo 1

17

Page 18: Akka streams scala italy2015

Akka Streams: Graph

• Source, Flow, and  Sink  are good for expressing linear computations. • But how to express a computation graph?

18

Page 19: Akka streams scala italy2015

Demo 2

19

Page 20: Akka streams scala italy2015

Akka Streams: Fan-out

• Broadcast  -­‐   given an input element emits to each output.

• Balance - given an input element emits to one of its output ports.

• UnZip - splits a stream of (A,B) tuples into two streams, one of type A and on of type B.

• FlexiRoute  -­‐ enables writing custom fan out elements using a simple DSL.

20

Page 21: Akka streams scala italy2015

Akka Streams: Fan-in

• Merge  -­‐   picks randomly from inputs pushing them one by one to its output. • MergePreferred  - like Merge but if elements are

available on preferred port, it picks from it, otherwise randomly from others.

•  ZipWith(fn)- takes a function of N inputs that given a value for each input emits 1 output element.

21

Page 22: Akka streams scala italy2015

Akka Streams: Fan-in cont’d

• Zip  - is a ZipWith specialised to zipping input streams of A and B into an (A,B) tuple stream. • Concat  - concatenates two streams (first consume

one, then the second one).

• FlexiMerge  - enables writing custom fan-in elements using a simple DSL.

22

Page 23: Akka streams scala italy2015

Demo 3

23

Page 24: Akka streams scala italy2015

What is back-pressure?

24

Page 25: Akka streams scala italy2015

Back-pressure? Example Without

Publisher[T] Subscriber[T]

25

Page 26: Akka streams scala italy2015

Back-pressure? Example Without

Fast Publisher Slow Subscriber

26

Page 27: Akka streams scala italy2015

Back-pressure? “Why would I need that!?”

27

Page 28: Akka streams scala italy2015

Back-pressure? Push + NACK model

28

Page 29: Akka streams scala italy2015

Back-pressure? Push + NACK model

Subscriber usually has some kind of buffer.

29

Page 30: Akka streams scala italy2015

Back-pressure? Push + NACK model

30

Page 31: Akka streams scala italy2015

Back-pressure? Push + NACK model

31

Page 32: Akka streams scala italy2015

Back-pressure? Push + NACK model

What if the buffer overflows?

32

Page 33: Akka streams scala italy2015

Back-pressure? Push + NACK model (a)

Use bounded buffer, drop messages + require re-sending

33

Page 34: Akka streams scala italy2015

Back-pressure? Push + NACK model (a)

Kernel does this! Routers do this!

(TCP)

Use bounded buffer, drop messages + require re-sending

34

Page 35: Akka streams scala italy2015

Back-pressure? Push + NACK model (b)Increase buffer size… Well, while you have memory available!

35

Page 36: Akka streams scala italy2015

Back-pressure? Push + NACK model (b)

36

Page 37: Akka streams scala italy2015

Negative ACKnowledgement

37

Page 38: Akka streams scala italy2015

Back-pressure? Example NACKing

Buffer overflow is imminent!

38

Page 39: Akka streams scala italy2015

Back-pressure? Example NACKingTelling the Publisher to slow down / stop sending…

39

Page 40: Akka streams scala italy2015

Back-pressure? Example NACKing

NACK did not make it in time, because M was in-flight!

40

Page 41: Akka streams scala italy2015

Back-pressure? NACKing is NOT enough.

41

Page 42: Akka streams scala italy2015

Back-pressure? !

speed(publisher) < speed(subscriber)

42

Page 43: Akka streams scala italy2015

Back-pressure? Fast Subscriber, No Problem

No problem!

43

Page 44: Akka streams scala italy2015

Back-pressure? Reactive-Streams

=

44

Page 45: Akka streams scala italy2015

Just push – not safe when Slow Subscriber !!

Just pull – too slow when Fast Subscriber

Back-pressure? RS: Dynamic Push/Pull

45

Page 46: Akka streams scala italy2015

!!!

Solution: Dynamic adjustment

Back-pressure? RS: Dynamic Push/Pull

Just push – not safe when Slow Subscriber !!

Just pull – too slow when Fast Subscriber

46

Page 47: Akka streams scala italy2015

Back-pressure? RS: Dynamic Push/PullSlow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.

47

Page 48: Akka streams scala italy2015

Back-pressure? RS: Dynamic Push/PullFast Publisher will send at-most 3 elements. This is pull-based-backpressure.

48

Page 49: Akka streams scala italy2015

Back-pressure? RS: Dynamic Push/Pull

Fast Subscriber can issue more Request(n), before more data arrives!

49

Page 50: Akka streams scala italy2015

Back-pressure? RS: Dynamic Push/PullFast Subscriber can issue more Request(n), before more data arrives. !Publisher can accumulate demand.

50

Page 51: Akka streams scala italy2015

Back-pressure? RS: Accumulate demand

Publisher accumulates total demand per subscriber.

51

Page 52: Akka streams scala italy2015

Back-pressure? RS: Accumulate demandTotal demand of elements is safe to publish. Subscriber’s buffer will not overflow.

52

Page 53: Akka streams scala italy2015

Demo 4

53

Page 54: Akka streams scala italy2015

Is that really all there is to know?

• Naaaa, there is a lot more for you to explore!

• If the existing building blocks are not enough, define your owns.

• Use mapAsync/mapAsyncUnordered for integrating with external services.

• Streams Error Handling.

• Handling TCP connections with Streams.

• Integration with Actors.

54

Page 55: Akka streams scala italy2015

What now?

• Use it:"com.typesafe.akka" %% "akka-stream-experimental" % "1.0-RC2" • Check out the Activator template

Akka Streams with Java8 or Scala.

• Akka Streams API doc and user guide for both Java8 and Scala.

• Code used for the demos https://github.com/dotta/akka-streams-demo/releases/tag/v02

55

Page 56: Akka streams scala italy2015

Next Steps

• Akka Streams 1.0 final soon. • Inclusion in future JDK (shooting for JDK9) • We aim at polyglot standard (JS, wire proto) • Try it out and give feedback! • http://reactive-streams.org/ • https://github.com/reactive-streams

56

Page 57: Akka streams scala italy2015

57

Page 58: Akka streams scala italy2015

©Typesafe 2015 – All Rights Reserved58