Intro to Akka Streams

streams

Agenda

• Reactive Streams

• Why Akka Streams?

• API Overview

Reactive Streams

public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}

public interface Subscriber<T> {

public void onSubscribe(Subscription s);

public void onNext(T t);

public void onError(Throwable t);

public void onComplete();}

public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {}

public interface Subscription {

public void request(long n);

public void cancel();}

Reactive Streams

A standardised spec/contract to achieve asynchronous

back-pressured stream processing.

Standardised ?

Gives us consistent interop between libraries and platforms that implement this spec.

everything is async & back-pressured

Reactive Streams

Stream API Stream API Stream API

Reactive Streams

Users use this API

Reactive Streams

Users use this API

Library authors use this API

Async?

• We know async IO from last week

• But there are other types of async operations, that cross over different async boundaries

• between applications

• between threads

• and over the network as we saw

Back-Pressured ?

Publisher[T] Subscriber[T]

Think abstractly about these lines.

“async boundary”

This can be the network, or threads on the same CPU.

What problem are we trying to solve?

Discrepancy in the rate of processing

• Fast Publisher / Slow Subscriber

• Slow Publisher / Fast Subscriber

Push Model

100 messages / 1 second

1 message / 1second

Fast Slow

drop overflowedrequire resending

has to keep trackof messages to resendnot safe & complicated

NACK ?

publisher didn’t receive NACK in timeso we lost that last message

not safe

Pull ?

100 messages / 1 second

1 message / 1second

FastSlow

gimme!

• Spam!

• Redundant messaging -> flooding the connection

• No buffer/batch support

A different approach

We have to take into account the following scenarios:

• Fast Pub / Slow Sub

• Slow Pub / Fast Sub

Which can happen dynamically

Demand(n)

Dynamic Push/Pull

bounded buffers with no overflowdemand can be accumulated

batch processing -> performance

• Cool let’s implement this using Actors!

• We can, it’s possible … but should it be done ?

The problem(s) with Akka Actors

Type Safety

Any => Unit

Composition

In FP this makes us warm and fuzzyval f: A => Bval g: B => C

val h: A => C = f andThen g

• Using Actors?

• An Actor is aware of who sent it messages and where it must forward/reply them.

• No compositionality without thinking about it explicitly.

Data Flow

• What are streams ? Flows of data.

• Imagine a 10 stage data pipeline you want to model

• Now imagine writing that in Actors.

• Following the flow of data in Actors requires jumping around all over the code base

• Low level, error prone and hard to reason about

Akka Streams APIbuilding blocks

Design Philosophy

• Everything we will cover now are blueprints that describe the actions/effects they perform.

• Reusability

• Compositionality

• “Design your program with a pure functional core,push side-effects to the end of the world and detonate to execute.

- some guy on stackoverflow

• Publisher of data

• Exactly one output

Image from boldradius.com

val singleSrc = Source.single(1)

val iteratorSrc = Source.fromIterator(() => Iterator from 0)

val futureSrc = Source.fromFuture(Future("abc"))

val collectionSrc = Source(List(1,2,3))

val tickSrc = Source.tick(initialDelay = 1 second,

interval = 1 second,tick = "tick-tock")

val requestSource = req.entity.dataBytes

• Subscriber (consumer) of data

• Describes where the data in our stream will go.

• Exactly one input

Sink.head

Sink.reduce[Int]((a, b) => a + b)

Sink.fold[Int, Int](0)(_ + _)

Sink.foreach[String](println)

FileIO.toPath(Paths.get("file.txt"))

val fold: Sink[Int, Future[Int]] = Sink.fold[Int, Int](0)(_ + _)

Input type

Materialized type

Available when the stream ‘completes’

val futureRes: Future[Int] = Source(1 to 10).runWith(fold)

futureRes.foreach(println)

So I can get data from somewhere

and I can put data somewhere else.

But I want to do something with it.

• A processor of data

• Has one input and one output

val double: Flow[Int, Int, NotUsed] = Flow[Int].map(_ * 2)

val src = Source(1 to 10)

val double = Flow[Int].map(_ * 2)

val negate = Flow[Int].map(_ * -1)

val print = Sink.foreach[Int](println)

val graph = src via double via negate to print

graph.run()

-2-4-6-8-10-12-14-16-18-20

• Flow is immutable, thread-safe, and thus freely shareable

• Are Linear flows enough ?

• No, we want to be able to describe arbitrarilly complex steps in our pipelines

Graphs

• We define multiple linear flows and then use the Graph DSL to connect them.

• We can combine multiple streams - fan in

• Split a stream into substreams - fan out

Fan-Out

Fan-In

A little example

Some sort of video uploading service

- Stream in video- Process it

- Store it

ByteStringConvert toArray[Byte]

flowbcast

Process HighRes flow

Process LowRes flow

Process MedRes flow

Sink.fromGraph(GraphDSL.create(highRes, mediumRes, lowRes)((_, _, _){ implicit b => (highSink, mediumSink, lowSink) => { import GraphDSL.Implicits._

val bcastInput = b.add(Broadcast[ByteString](1)) val bcastRawBytes = b.add(Broadcast[Array[Byte]](3)) val processHigh: Flow[Array[Byte], ByteString, NotUsed] val processMedium: Flow[Array[Byte], ByteString, NotUsed] val processLow: Flow[Array[Byte], ByteString, NotUsed]

bcastInput.out(0) ~> byteAcc ~> bcastRawBytes ~> processHigh ~> highSink bcastRawBytes ~> processMedium ~> mediumSink bcastRawBytes ~> processLow ~> lowSink

SinkShape(bcastInput.in) }})

Our custom Sink

Has one input of type ByteString

Takes 3 Sinks, which can be Files, DBs, etc.

Describes 3 processing stagesThat are Flows of Array[Byte] => ByteString

Emits result to the 3 Sinks

Has a type of:Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]

Sink[ByteString, (Future[IOResult], Future[IOResult], Future[IOResult])]

Materialized values

Things we didn’t have time for

• Integrating with Actors

• Buffering and throttling streams

• Defining custom Graph shapes and stages

Thanks for listening!

Intro to Akka Streams

Engineering

Transcript of Intro to Akka Streams

A reactive platform for real-time asset trading case study ...reactive+platform...scala, akka persistence, akka cluster, akka streams, cassandra (dse), apache kafka, gatling, prometheus,

Kiki, principal enterprise architect @Lightbend · Akka Streams Akka Cluster Akka Http. Scalability Cube: 3 dimensions of scalability Y-axis: scale via functional compositi on x-axis:

Akka Streams - From Zero to Kafka

Fresh from the Oven (04.2015): Experimental Akka Typed and Akka Streams

TDC2017 | POA Trilha Programacao Funcional - Reactive Streams com Akka Streams

Reactive Streams in the Web (GOTO Berlin)...Reactive Streams Reactive Extensions Marble Diagrams RxJava Reactor Akka Streams Backpressure Declarative Programming Higher-Order Functions

A dive into akka streams: from the basics to a real-world scenario

Asynchronous stream processing with Akka Streams

Journey into Reactive Streams and Akka Streams

Introduction to Akka-Streams

Akka - A Brief Intro

Building Kafka-based Microservices with Akka Streams and ... · Akka Streams, Kafka Streams - libraries for “data-centric micro services”. Smaller scale, but great flexibility.

End to End Akka Streams / Reactive Streams - from Business to Socket

Akka HTTP: The Reactive Web Toolkitdownloads.typesafe.com/.../ScalaDaysSF2015/T1_Kuhn_Akka_Strea… · • reactive-streams 1.0.0-RC3 • Akka Streams & HTTP 1.0-M4 • still missing:

Scala usergroup stockholm - reactive integrations with akka streams

Lessons Learned From PayPal: Implementing Back-Pressure With Akka Streams And Kafka

Akka Streams and Bloom Filters - Data Science€¦ · Akka Streams • Built on Akka • “…move data across an asynchronous boundary, without loss, buffering or resource exhaustion.”

Akka Streams and Bloom Filters€¦ · Akka Streams and Bloom Filters Or: How I learned to stop worrying and love resilient elastic distributed real-time ... ApproximateBoolean (false,1.0

Mirco Dotta - Akka Streams

How to manage large amounts of data with akka streams