akka streams,
Exploring Reactive Integrations with
Alpakka and Kafka
Konrad `ktoso` Malawski
Konrad `ktoso` MalawskiAkka Team, Reactive Streams TCK, Persistence, HTTP, Remoting / Cluster
Make building powerful concurrent & distributed applications simple.Akka is a toolkit and runtime for building highly concurrent, distributed, and resilient message-driven applications on the JVM
Actors – simple & high performance concurrency Cluster / Remoting – location transparency, resilience Cluster tools – and more prepackaged patterns Streams – back-pressured stream processing Persistence – Event Sourcing HTTP – complete, fully async and reactive HTTP Server Official Kafka, Cassandra, DynamoDB integrations, tons more in the community
Complete Java & Scala APIs for all features
What’s in the toolkit?
“Stream”has many meanings
akka streamsAsynchronous back pressured stream processing
Source Sink
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
(possible) asynchronous
boundaries
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
OutOfMemoryError!!
Flow
akka streamsAsynchronous back pressured stream processing
Source Sink
10 msg/s 1 msg/s
hand me 3 morehand me 3 more
1 msg/s Flow
akka streamsNot only linear streams
Source
SinkFlow
SourceSink
FlowFlow
And the many meanings it carries.
Reactive
The many meanings of Reactive
“Not-quite-Reactive-System” The reason we started researching into transparent to users flow control.
Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols
http://www.reactive-streams.org
Reactive StreamsA buiding-block of Reactive Systems, not the “entire story”.
Reactive StreamsReactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure. This encompasses efforts aimed at runtime environments as well as network protocols
http://www.reactive-streams.org
Part of JDK 9java.util.concurrent.Flow
http://openjdk.java.net/projects/jdk9/
JEP-266 – soon…!public final class Flow { private Flow() {} // uninstantiable
@FunctionalInterface public static interface Publisher<T> { public void subscribe(Subscriber<? super T> subscriber); }
public static interface Subscriber<T> { public void onSubscribe(Subscription subscription); public void onNext(T item); public void onError(Throwable throwable); public void onComplete(); }
public static interface Subscription { public void request(long n); public void cancel(); }
public static interface Processor<T,R> extends Subscriber<T>, Publisher<R> { }}
Reactive Streams
RS Library A RS library B
async boundary
Reactive Streams
RS Library A RS library B
async boundary
Make building powerful concurrent & distributed applications simple.
The APIAkka Streams
Complete and awesome Java and Scala APIs (Just like everything in Akka)
Akka Streams in 20 seconds:
Source<Integer, NotUsed> source = null; Flow<Integer, String, NotUsed> flow = Flow.<Integer>create().map((Integer n) -> n.toString()); Sink<String, CompletionStage<Done>> sink = Sink.foreach(str -> System.out.println(str)); RunnableGraph<NotUsed> runnable = source.via(flow).to(sink);runnable.run(materializer);
Akka Streams in 20 seconds:
CompletionStage<String> firstString = Source.single(1) .map(n -> n.toString()) .runWith(Sink.head(), materializer);
Source.single(1).map(i -> i.toString).runWith(Sink.head())
// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>
Akka Streams in 20 seconds:
Source.single(1).map(i -> i.toString).runWith(Sink.head())
// types: _Source<Int, NotUsed> Flow<Int, String, NotUsed> Sink<String, CompletionStage<String>>
Akka Streams in 20 seconds:
What is “materialization” really?
What is “materialization” really?
What is “materialization” really?
Introspection, and more coming…
Highly sophisticated stream introspection capabilities.
AlpakkaA community for Streams connectors
http://blog.akka.io/integrations/2016/08/23/intro-alpakka
Alpakka – a community for Stream connectors
Threading & Concurrency in Akka Streams Explained (part I)
Mastering GraphStages (part I, Introduction)
Akka Streams Integration, codename Alpakka
A gentle introduction to building Sinks and Sources using GraphStage APIs (Mastering GraphStages, Part II)
Writing Akka Streams Connectors for existing APIs
Flow control at the boundary of Akka Streams and a data provider
Akka Streams Kafka 0.11
Alpakka – a community for Stream connectors
http://developer.lightbend.com/docs/alpakka/current/
Alpakka – a community for Stream connectors
Demo
Akka Streams & HTTP
streams& HTTP
A core feature not obvious to the untrained eye…!
Akka Streams / HTTP
Quiz time! TCP is a ______ protocol?
A core feature not obvious to the untrained eye…!
Akka Streams / HTTP
Quiz time! TCP is a STREAMING protocol!
Streaming in Akka HTTP
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
HttpServer as a: Flow[HttpRequest, HttpResponse]
Streaming in Akka HTTP
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Streaming in Akka HTTP
HttpServer as a: Flow[HttpRequest, HttpResponse]
HTTP Entity as a: Source[ByteString, _]
Websocket connection as a: Flow[ws.Message, ws.Message]
http://doc.akka.io/docs/akka/2.4/scala/stream/stream-customize.html#graphstage-scala “Framed entity streaming”
http://doc.akka.io/docs/akka/2.4/java/http/routing-dsl/source-streaming-support.html
Resource efficiency and dynamic fan-out
Resource efficiency and dynamic fan-out
Streaming from Akka HTTP
Streaming from Akka HTTP
Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);
final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));
final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );
final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");
}
Streaming from Akka HTTP (Java) public static void main(String[] args) { final ActorSystem system = ActorSystem.create(); final Materializer materializer = ActorMaterializer.create(system); final Http http = Http.get(system);
final Source<Tweet, NotUsed> tweets = Source.repeat(new Tweet("Hello world"));
final Route tweetsRoute = path("tweets", () -> completeWithSource(tweets, Jackson.marshaller(), EntityStreamingSupport.json()) );
final Flow<HttpRequest, HttpResponse, NotUsed> handler = tweetsRoute.flow(system, materializer);
http.bindAndHandle(handler, ConnectHttp.toHost("localhost", 8080), materializer ); System.out.println("Running at http://localhost:8080");
}
Streaming from Akka HTTP (Scala)object Example extends App with SprayJsonSupport with DefaultJsonProtocol { import akka.http.scaladsl.server.Directives._ implicit val system = ActorSystem() implicit val mat = ActorMaterializer()
implicit val jsonRenderingMode = EntityStreamingSupport.json() implicit val TweetFormat = jsonFormat1(Tweet)
def tweetsStreamRoutes = path("tweets") { complete { Source.repeat(Tweet("")) } } Http().bindAndHandle(tweetsStreamRoutes, "127.0.0.1", 8080) System.out.println("Running at http://localhost:8080");}
Next steps for Akka
Completely new Akka Remoting (goal: 700.000+ msg/s (!)), (it is built using Akka Streams, Aeron).
More integrations for Akka Streams stages, project Alpakka.
Reactive Kafka polishing with SoftwareMill, Krzysiek Ciesielski
Akka Typed progressing again, likely towards 3.0.
Akka HTTP 2.0 Proof of Concept in progress.
Collaboration with Reactive Sockets
Ready to adopt on prod?
Totally, go for it.
Akka <3 contributionsEasy to contribute tickets:
https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3Aeasy-to-contribute https://github.com/akka/akka/issues?q=is%3Aissue+is%3Aopen+label%3A%22nice-to-have+%28low-prio%29%22
Akka Stream Contrib https://github.com/akka/akka-stream-contrib
Mailing list: https://groups.google.com/group/akka-user
Public chat rooms: http://gitter.im/akka/dev developing Akka http://gitter.im/akka/akka using Akka
More resources:
Read more tutorials and deep-dives:
http://blog.akka.io/ https://www.lightbend.com/resources
Reactive PlatformReactive Platform
Reactive Platform
Further reading:
Reactive Streams: reactive-streams.org Akka documentation: akka.io/docs Free O’Reilly report – bit.ly/why-reactive
Example Sources: ktoso/akka-streams-alpakka-talk-demos-2016
Contact: Konrad [email protected] Malawski http://kto.so / @ktosopl
Top Related