AKKA STREAMSFROM ZERO TO KAFKACreated by / Mark Harrison @markglh
HOW IT ALL BEGAN“Reactive Streams is an initiative to provide astandard for asynchronous stream processing
with non-blocking back pressure. Thisencompasses efforts aimed at runtime
environments (JVM and JavaScript) as well asnetwork protocols.”
WHYEf�ciently processing large indeterminate streams is hardAvoiding blocking is essential to maximise performanceEvery stage in the stream needs to be able to push and pullWe don't want to overload (or starve!) downstreamconsumers...
HOWTreat data as a stream of elementsAsynchronous non-blocking data and demand �owsDemand �ows upstream, causing data to �ow downstreamData �ow is therefore restricted by demand
Back Pressure!!Demand happens on a separate �ow!
WHATThe Reactive Streams speci�cation is just that
A collection of interfaces methods and protocolsProvides example implementations and a TCK forveri�cationAimed at providing a way to build commonimplementations
INTRODUCING AKKA STREAMS!!AKKA'S IMPLEMENTATION OF REACTIVE STREAMS
DESIGN PRINCIPLESExplicitness over magic (I'm looking at you Shapeless!)Fully composable
Each component, or set of componenents can be combinedEach building block is immutableFully compatible with other Reactive Stream implementations
BUILDING BLOCKS
BUILDING BLOCKS CONT...Source
Traditionally known as a producerSupplies messages that will �ow downstreamExactly one output stream
SinkTraditionally known as a consumerEnd point of the stream, this is where messages end up
BUILDING BLOCKS CONT...Flow
A processing stage in the StreamUsed to compose StreamsExactly one input and one output streamSee also BidirectionalFlow (two in -> two out)
BUILDING BLOCKS CONT...RunnableGraphs
A pre-assembled set of Stream components, packaged intoa Graph.All exposed ports are connected (between a Source andSink)This can then be Materialized
BUILDING BLOCKS CONT...Composite Flows
It is possible to wrap several components into morecomplex onesThis composition can then be treated as one block
Partial Flow GraphsAn incomplete Flow (Graph)Can be used to construct more complex Graphs easily
BUILDING BLOCKS CONT...Materializer
Once complete, the �ow is Materialized in order to startstream processingSupports fully distributed stream processing
Each step must be either serializable immutable valuesor ActorRefs
Fails immediately at runtime if the Graph isn't complete
ERRORS VS FAILURESErrors handlied within the stream as normal data elements
Passed using the onNext functionFailure means that the stream itself has failed and is collapsing
Raises the onError signal... (???)Each block in the �ow can choose to absorb or propagate theerrors
Possibly resulting the the complete collapse of the �ow
FIRST THINGS FIRSTWe need to create an ActorSystem and Materializer
implicit val system = ActorSystem("actors") implicit val materializer = ActorMaterializer()
SIMPLE STREAMWe need to create an ActorSystem and Materializer
Source(1 to 5) .filter(_ < 3) // 1, 2 .map(_ * 2) // 2, 4 .to(Sink.foreach(println)) .run() //prints 2 4
COMPOSING ELEMENTS TOGETHERWe can combine multiple components together
Composing elements together val nestedSource = Source(1 to 5) .map(_ * 2) val nestedFlow = Flow[Int] .filter(_ <= .map(_ + 2) val sink = Sink.foreach(println) //link up the Flow to a Sink val nestedSink = nestedFlow.to(Sink.foreach(println)) // Create a RunnableGraph - and run it! Prints 4 6 nestedSource.to(nestedSink).run()
COMPOSING ELEMENTS TOGETHER CONT...Alternatively we could do this, linking them in one step
nestedSource .via(nestedFlow) .to(Sink.foreach(println(_)))
COMPOSING ELEMENTS TOGETHER CONT...
GRAPH PROCESSING STAGESFan OutBroadcast[T] – (1 input, N outputs)Balance[T] – (1 input, N outputs)...
Fan InMerge[In] – (N inputs , 1 output)...
Timer DrivengroupedWithin(Int, Duration)
Groups elements when either the number or duration isreached (whichever is �rst). Very useful for batchingmessages.
See the Akka Stream docs for more!
GRAPH PROCESSING STAGES CONT...
THE GRAPH DSLWhenever you want to perform multiple operations tocontrol the Flow of a Graph, manually constructing them asabove can become very clumbersome and tedius, not tomentioned hard to maintain.For this reason the Akka team have written a DSL to helpwrite complex Graphs.
THE GRAPH DSLval g = FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => //This provides the DSL import FlowGraph.Implicits._ val in = Source(1 to 3) val out = Sink.foreach(println) //2 outputs, 2 inputs val bcast = builder.add(Broadcast[Int](2)) val merge = builder.add(Merge[Int](2)) val f1, f2, f3, f4 = Flow[Int].map(_ + 10) in ~> f1 ~> bcast ~> f2 ~> merge ~> f3 ~> out bcast ~> f4 ~> merge } g.run() //Prints 31 31 32 32 33 33
THE GRAPH DSL CONT...
EXAMPLE - REACTIVE KAFKAThe guys at SoftwareMill have implemented a wrapper forApache Kafka
Tried and tested by yours trulyhttps://github.com/softwaremill/reactive-kafka
EXAMPLE - REACTIVE KAFKA CONT...Source is a Kafka ConsumerSink is a Kafka Publisher
val kafka = new ReactiveKafka() val publisher: Publisher[StringKafkaMessage] = kafka.consume( ConsumerProperties(...) ) val subscriber: Subscriber[String] = kafka.publish( ProducerProperties(...) ) Source(publisher).map(_.message().toUpperCase) .to(Sink(subscriber)).run()
A REAL WORLD EXAMPLE
A REAL WORLD EXAMPLE CONT...FlowGraph.closed() { implicit builder: FlowGraph.Builder[Unit] => import FlowGraph.Implicits._ val in = Source(kafkaConsumer) val out = Sink.foreach(println) val bcast = builder .add(Broadcast[StringKafkaMessage](2)) val merge = builder .add(Merge[StringKafkaMessage](2)) val parser1, parser2 = Flow[StringKafkaMessage] .map(...) val group = Flow[StringKafkaMessage].grouped(4) in ~> bcast ~> parser1 ~> merge ~> group ~> out bcast ~> parser2 ~> merge }.run()
IT'S BEEN EMOTIONAL...Slides at
Follow me
http://markglh.github.io/AkkaStreams-Madlab-Slides
@markglh
Top Related