Reactive Kafka with Akka Streams

Krzysztof CiesielskiReactive Summit 2016, Austin, TX

about.me

• Scala developer @

• Editor of ScalaTimes

• Open Source (akka-stream-kafka, scala-clippy and other)

Premier Consulting Partner

akka-stream-kafka

• Formely reactive-kafka

• A library for accessing Kafka as stages of Akka Streams

Apache Kafkaa message log

8 7 6 5 4 3 2 1

Producer

Producer

Consumer

Consumer

topic “topic1”

Apache Kafkaa distributed message log

4 3 2 1

Producer

4 3 2 1

partition 1

partition 2

“topic1”

“topic1”

Apache Kafkaa distributed message log

Consumer

balancing4 3 2 1

Producer

4 3 2 1

partition 1

partition 2

“topic1”

“topic1”

Apache Kafkaa distributed message log

Consumer1

Consumer2

4 3 2 1

Producer

4 3 2 1

partition 1

partition 2

“topic1”

“topic1”

Apache Kafkaa distributed message log

4 3 2 1

4 3 2 1

partition 1

partition 3

“topic1”

“topic1”

Consumer1

Consumer2

4 3 2 1

partition 2“topic1”manual assignment

Apache KafkaUses Zookeeper

Leader election

Cluster coordination

Shared configuration

Apache Kafka

Decoupling of producers and consumers

Apache KafkaCommit

8 7 6 5 4 3 2 1 Consumer

commit(p1, o3)

msg1, msg2

8 7 6 5 4 3 2 1 Consumer

Apache KafkaCommit

Apache KafkaCommit

8 7 6 5 4 3 2 1 New consumermsg3

Akka Streams

• DSL for describing data transformation pipelines.

• Uses Actor Model.

• Implements the Reactive Streams specification.

Akka Streams

• Focused on backpressure and async processing.

• Comes with a powerful test kit.

• Extensible.

AlpakkaInitiative to build a rich set of connectors

HTTP

Streaming TCP

Streaming File IO

Reactive Streams (MongoDB Reactive Streams Java Driver)

Akka Streams + Kafka

Akka Streams + Kafka

Akka Streams + Kafka

Akka Streams + Kafka

akka-stream-kafka

• Started as reactive-kafka @ SoftwareMill

• First users

• Alexey Romanchuk has rebuilt the library

• Akka Team joined (Patrik Nordwall, Endre Varga)

Example: plain consumer

Example: plain consumer

Example: plain consumer

+async processing

+error handling

+backpressure

Consumer Actor

SourceStage

ConsumerActorconsumer (Kafka API)

RequestMessages

demand messages

Messages

Scheduled Poll

poll()

dispatcher

Plain consumer performance

1,202,081944,746

Apache KafkaCommit

8 7 6 5 4 3 2 1 Consumer

commit(p1, o3)

msg1, msg2

Consumer Actor and commit

SourceStage

ConsumerActorconsumer (Kafka API)

? Commit

commit()

Scheduled Poll

poll()

dispatcher

msg

msg

commitAsync()

callback

Request Messages

Committable Source

Committable Source

Batched Committable Source

Batched Committable Source

At-least-once delivery

456,695

1,194,331

External commit source

Producer

Producer Vanilla API

Producer as a Flow

Producer as a Flow

From Kafka to Kafka

From Kafka to KafkaSourceStage

Producer Stage

msg send()

callback()msg

msg mapAsync

ConsumerActor

From Kafka to Kafka

It’s streaming streams

backpressure per partition

Source[(TopicPartition, Source[CommittableMessage[Array[Byte], String], NotUsed]), Control]

It’s streaming streams

Error handling

Java API

Kafka Streams

Kafka Streamsakka-stream-kafka

Kafka Streams

Scala + Java, backpressure, async processing, interop with Akka Streams

Java, stateful processors, windowing, joining, aggregation operations

Thank you!

Join us!https://gitter.im/akka/reactive-kafka

@kpciesielski