Kafka & message busRobin GRAILLON & Alexandre ANDRÉ

12/10/2016

Synchronous

● Call● Processing

○ Might be long● Response

Asynchronous

● Call● Response● Post-Processing

Message Bus

● Events aggregation● CQRS oriented● µS oriented● Event sourcing oriented● Language agnostic● Multiple implementations

○ RabbitMQ○ Kafka○ etc.

Apache Kafka

● LinkedIn creation● Open sourced in 2011● Filesystem oriented● Wrote in Scala● Highly scalable● Used by big companies

○ LinkedIn○ Netflix○ Spotify○ Meetic

LinkedIn statistics

● 800 billion m/day○ 175 TB

● 13 million m/sec○ 2.75 GB

● 1100 kafka instances○ 60 clusters

Basic workflow

Producer 1

Producer 2

Producer 3

Producer X

Consumer 1

Consumer 2

Consumer 3

Consumer Y

Kafka Stack

Kafka stack

● Zookeeper : scalability, manager● Kafka broker : kafka server instance● Consumer : consumes events and do things● Producer : produces events (like user as connected)● topic : event name● partition : way to split messages between brokers

Kafka partitions (1)

For one topic (let’s say “astronaut.connection”)

Kafka stack & partitions

Let’s play...

...but easily !./console run

HELPrun:zk run zookeeperkill:zk kill zookeeperrun:kk run kafkakill:kk kill kafkaproducer run producer on topicconsumer run consumer on topict:c create a topict:d delete a topict:l list topicsWhat do you want to do?

Step 1: launch Zookeeper!

What do you want to do? run:zkConfiguration file? [./zookeeper.properties] Port? [2181] ./../bin/zookeeper-server-start.sh ./zookeeper.propertiesWhat do you want to do?

Step 2: launch Kafka!

What do you want to do? run:kk

Step 3: list topics!

What do you want to do? t:lZookeeper host? [localhost:2181] ./../bin/kafka-topics.sh --list --zookeeper localhost:2181

What do you want to do?

Step 4: create a topic!

What do you want to do? t:cTopic name? [test] Zookeeper host? [localhost:2181] Partitions? [1] Replication factor? [1] ./../bin/kafka-topics.sh --create --zookeeper localhost:2181 --topic test --partitions 1 --replication-factor 1Created topic "test".What do you want to do?

Step 5: list topics AGAIIIIN!

What do you want to do? t:l Zookeeper host? [localhost:2181] ./../bin/kafka-topics.sh --list --zookeeper localhost:2181test

What do you want to do?

Step 6: run a consumer!

New terminal !

$ ./console run

What do you want to do? consumerTopic name? [test] Zookeeper host? [localhost:2181] From beginning? [1] ./../bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

Step 7: generate data!

What do you want to do? gMessage? [I'm an astronaut!] How many times? [1] 10000000File? [data_10000000.txt] 10000000/10000000 [============================] 100%

Step 8: produce events!

Back to the other terminal

What do you want to do? pTopic name? [test] brokers? [localhost:9092] data_10000000.txtDataset to use? [data_10000000.txt] ./../bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < ./data/data_10000000.txt

GroupId: experiment 1

● 2 groupId● 2 partitions● 2 consumers

GroupId: experiment 2

● 1 groupId● 2 consumer

Documentation/resources

● Kafka quick setuphttps://kafka.apache.org/quickstart

● Kafka at LinkedInhttps://engineering.linkedin.com/kafka/running-kafka-scale

● Why zookeeper ?https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka

Titre 1 Titre 2