Post on 16-Apr-2017
about.me
• Scala developer @
• Editor of ScalaTimes
• Open Source (akka-stream-kafka, scala-clippy and other)
Premier Consulting Partner
akka-stream-kafka
• Formely reactive-kafka
• A library for accessing Kafka as stages of Akka Streams
Apache Kafkaa distributed message log
4 3 2 1
Producer
4 3 2 1
partition 1
partition 2
“topic1”
“topic1”
Apache Kafkaa distributed message log
Consumer
balancing4 3 2 1
Producer
4 3 2 1
partition 1
partition 2
“topic1”
“topic1”
Apache Kafkaa distributed message log
Consumer1
Consumer2
4 3 2 1
Producer
4 3 2 1
partition 1
partition 2
“topic1”
“topic1”
Apache Kafkaa distributed message log
4 3 2 1
4 3 2 1
partition 1
partition 3
“topic1”
“topic1”
Consumer1
Consumer2
4 3 2 1
partition 2“topic1”manual assignment
Akka Streams
• DSL for describing data transformation pipelines.
• Uses Actor Model.
• Implements the Reactive Streams specification.
Akka Streams
• Focused on backpressure and async processing.
• Comes with a powerful test kit.
• Extensible.
AlpakkaInitiative to build a rich set of connectors
HTTP
Streaming TCP
Streaming File IO
Reactive Streams (MongoDB Reactive Streams Java Driver)
akka-stream-kafka
• Started as reactive-kafka @ SoftwareMill
• First users
• Alexey Romanchuk has rebuilt the library
• Akka Team joined (Patrik Nordwall, Endre Varga)
Consumer Actor
SourceStage
ConsumerActorconsumer (Kafka API)
RequestMessages
demand messages
Messages
Scheduled Poll
poll()
dispatcher
Consumer Actor and commit
SourceStage
ConsumerActorconsumer (Kafka API)
? Commit
commit()
Scheduled Poll
poll()
dispatcher
msg
msg
commitAsync()
callback
Request Messages
Source[(TopicPartition, Source[CommittableMessage[Array[Byte], String], NotUsed]), Control]
It’s streaming streams
Kafka Streamsakka-stream-kafka
Kafka Streams
Scala + Java, backpressure, async processing, interop with Akka Streams
Java, stateful processors, windowing, joining, aggregation operations
Thank you!
Join us!https://gitter.im/akka/reactive-kafka
@kpciesielski