Reactive integrations with Akka Streams

52
akka streams Reactive Integrations with that just workJohan Andrén Konrad Malawski

Transcript of Reactive integrations with Akka Streams

Page 1: Reactive integrations with Akka Streams

akka streamsReactive Integrations with

that just work™

Johan Andrén Konrad Malawski

Page 2: Reactive integrations with Akka Streams

Johan AndrénAkka Team Stockholm Scala User Group

Page 3: Reactive integrations with Akka Streams

Konrad `ktoso` MalawskiAkka Team, Reactive Streams TCK, Persistence, HTTP

Page 4: Reactive integrations with Akka Streams

Make building powerful concurrent & distributed applications simple.Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM

Page 5: Reactive integrations with Akka Streams

Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community

Complete Java & Scala APIs for all features

What’s in the toolkit?

Page 6: Reactive integrations with Akka Streams
Page 7: Reactive integrations with Akka Streams

“Stream”has many meanings

Page 8: Reactive integrations with Akka Streams

akka streamsAsynchronous back pressured stream processing

Source Sink

Flow

Page 9: Reactive integrations with Akka Streams

akka streamsAsynchronous back pressured stream processing

Source Sink

(possible) asynchronous

boundaries

Flow

Page 10: Reactive integrations with Akka Streams

akka streamsAsynchronous back pressured stream processing

Source Sink

10 msg/s 1 msg/s

OutOfMemoryError!!

Flow

Page 11: Reactive integrations with Akka Streams

akka streamsAsynchronous back pressured stream processing

Source Sink

10 msg/s 1 msg/s

hand me 3 morehand me 3 more

1 msg/s Flow

Page 12: Reactive integrations with Akka Streams

akka streamsNot only linear streams

Source

SinkFlow

SourceSink

FlowFlow

Page 13: Reactive integrations with Akka Streams

Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols

http://www.reactive-streams.org

Page 14: Reactive integrations with Akka Streams

Part of JDK 9java.util.concurrent.Flow

Page 15: Reactive integrations with Akka Streams

Reactive Streams

RS Library A RS library B

async boundary

Page 16: Reactive integrations with Akka Streams

Reactive Streams

RS Library A RS library B

async boundary

Make building powerful concurrent & distributed applications simple.

Page 17: Reactive integrations with Akka Streams

The APIAkka Streams

Complete and awesome Java and Scala APIs (Just like everything in Akka)

Page 18: Reactive integrations with Akka Streams

Akka Streams in 20 seconds:

Source<Integer, NotUsed> source = null; Flow<Integer, String, NotUsed> flow = Flow.<Integer>create().map((Integer n) -> n.toString()); Sink<String, CompletionStage<Done>> sink = Sink.foreach(str -> System.out.println(str)); RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);runnable.run(materializer);

Page 19: Reactive integrations with Akka Streams

Akka Streams in 20 seconds:

CompletionStage<String> firstString = Source.single(1) .map(n -> n.toString()) .runWith(Sink.head(), materializer);

Page 20: Reactive integrations with Akka Streams

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Akka Streams in 20 seconds:

Page 21: Reactive integrations with Akka Streams

Source.single(1).map(i -> i.toString).runWith(Sink.head())

// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>

Akka Streams in 20 seconds:

Page 22: Reactive integrations with Akka Streams

Materialization

Gears from GeeCON.org,(it’s an awesome conf)

Page 23: Reactive integrations with Akka Streams

What is “materialization” really?

Page 24: Reactive integrations with Akka Streams

What is “materialization” really?

Page 25: Reactive integrations with Akka Streams

What is “materialization” really?

Page 26: Reactive integrations with Akka Streams

What is “materialization” really?

Page 27: Reactive integrations with Akka Streams

What is “materialization” really?

Check out the “Implementing an akka-streams materializer for big data”

talk later today.

Page 28: Reactive integrations with Akka Streams

AlpakkaA community for Streams connectors

http://blog.akka.io/integrations/2016/08/23/intro-alpakka

Page 29: Reactive integrations with Akka Streams

Alpakka – a community for Stream connectors

Threading & Concurrency in Akka Streams Explained (part I)

Mastering GraphStages (part I, Introduction)

Akka Streams Integration, codename Alpakka

A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II)

Writing Akka Streams Connectors for existing APIs

Flow control at the boundary of Akka Streams and a data provider

Akka Streams Kafka 0.11

Page 30: Reactive integrations with Akka Streams

Alpakka – a community for Stream connectors

Existing examples:MQTT AMQP

Streaming HTTPStreaming TCP

Streaming FileIOCassandra Queries

“Reactive Kafka” (akka-stream-kafka)S3, SQS & other Amazon APIs

Streaming JSONStreaming XML

Page 31: Reactive integrations with Akka Streams

Alpakka – a community for Stream connectors

Demo

Page 32: Reactive integrations with Akka Streams

Alpakka – a community for Stream connectors

Demo

Page 33: Reactive integrations with Akka Streams

Akka Streams & HTTP

streams& HTTP

Page 34: Reactive integrations with Akka Streams

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a ______ protocol?

Page 35: Reactive integrations with Akka Streams

A core feature not obvious to the untrained eye…!

Akka Streams / HTTP

Quiz time! TCP is a STREAMING protocol!

Page 36: Reactive integrations with Akka Streams

Streaming in Akka HTTP

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

HttpServer as a: Flow[HttpRequest, HttpResponse]

Page 37: Reactive integrations with Akka Streams

Streaming in Akka HTTP

HttpServer as a: Flow[HttpRequest, HttpResponse]

HTTP Entity as a: Source[ByteString, _]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Page 38: Reactive integrations with Akka Streams

Streaming in Akka HTTP

HttpServer as a: Flow[HttpRequest, HttpResponse]

HTTP Entity as a: Source[ByteString, _]

Websocket connection as a: Flow[ws.Message, ws.Message]

http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”

http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html

Page 39: Reactive integrations with Akka Streams

It’s turtles buffers all the way down!

xkcd.com

Page 40: Reactive integrations with Akka Streams

Streaming from Akka HTTP

Page 41: Reactive integrations with Akka Streams

Streaming from Akka HTTP

Page 42: Reactive integrations with Akka Streams

Streaming from Akka HTTP

No demand from TCP=

No demand upstream=

Source won’t generate tweets

=>

Bounded memory stream processing!

Demo

Page 43: Reactive integrations with Akka Streams

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

}

Page 44: Reactive integrations with Akka Streams

Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);

final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));

final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );

final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);

http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");

}

Page 45: Reactive integrations with Akka Streams

Streaming from Akka HTTP (Scala)object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer()

implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet)

def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080");}

Page 46: Reactive integrations with Akka Streams

Next steps for Akka

Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron).

More integrations for Akka Streams stages, project Alpakka.

Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski

Akka Typed progressing again, likely towards 3.0.

Akka HTTP 2.0 Proof of Concept in progress.

Collaboration with Reactive Sockets

Page 47: Reactive integrations with Akka Streams

Ready to adopt on prod?

Page 48: Reactive integrations with Akka Streams

Totally, go for it.

Page 49: Reactive integrations with Akka Streams

Akka <3 contributionsEasy to contribute tickets:

https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22

Akka Stream Contrib https://github.com/akka/akka-stream-contrib

Mailing list: https://groups.google.com/group/akka-user

Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka

Page 50: Reactive integrations with Akka Streams

Reactive PlatformReactive Platform

Reactive Platform

Page 51: Reactive integrations with Akka Streams

Further reading:Reactive Streams: reactive-streams.org Akka documentation: akka.io/docs Free O’Reilly report – very out soon.

Example Sources: ktoso/akka-streams-alpakka-talk-demos-2016

Get involved: sources: github.com/akka/akka mailing list: akka-user @ google groups gitter channel: https://gitter.im/akka/akka

Contact: Konrad [email protected] Malawski http://kto.so / @ktosopl