Enterprise Kafka: Kafka as a Service

SITE RELIABILITY ENGINEERING©2014 LinkedIn Corporation. All Rights Reserved.

Enterprise KafkaKafka as a Service

SITE RELIABILITY ENGINEERING©2014 LinkedIn Corporation. All Rights Reserved. 2

Why Am I Here?

You want to find out what this “Kafka” thing is

You’re running Kafka, but you want to go big

You’re looking for some neat whizbangs

SITE RELIABILITY ENGINEERING©2014 LinkedIn Corporation. All Rights Reserved.

Clark HaskinsSite Reliability EngineerLinkedIn

Todd PalinoSite Reliability EngineerLinkedIn


Who Are We?

Kafka SRE at LinkedIn

Site Reliability Engineering– Administrators– Architects– Developers

Keep the site running, always


Kafka Overview


What Is Kafka?


What Is Kafka?

Broker AP0

AP1

AP1

AP0 AP0

Consumer

Producer

Zookeeper


Attributes of a Kafka Cluster

Disk Based

Durable

Scalable

Low Latency

Finite Retention

NOT Idempotent (yet)


Kafka At LinkedIn

Multiple Datacenters, Multiple Clusters

Mirroring between clusters

Message Types– Metrics– Tracking– Queuing

Data transport from applications to Hadoop, and back


Kafka At LinkedIn


Kafka At LinkedIn

300+ Kafka brokers Over 18,000 topics 140,000+ Partitions

220 Billion messages per day 40 Terabytes In 160 Terabytes Out

Peak Load– 3.25 Million messages per second– 5.5 Gigabits/sec Inbound– 18 Gigabits/sec Outbound


Challenges We Have Overcome


Solutions

Kafka is young…..we Influenced development

Operations wizardry…


Hyper Growth

Need to expand clusters to keep up with site traffic, and then balance them.


Adding brokers

Brokers

Consumers

Producers

AP1

AP0

BP1

BP0

aP5

AP4

BP5

BP4

AP3

AP2

BP3

BP2

AP7

AP6

BP7

BP6

AP5

AP4

BP5

BP4

AP1

AP0

BP1

BP0

AP7

AP6

BP7

BP6

AP3

AP2

BP3

BP2

CP1

CP0

CP3

CP2

CP1

CP0

CP3

CP2


Adding a broker(with broker leveling)

Brokers

Consumers

Producers

AP1

AP0

BP1

BP0

AP5

AP4

BP5

BP4

AP3

AP2

BP3

BP2

AP7

AP6

BP7

BP6

AP5

AP4

BP5

BP4

AP1

AP0

BP1

BP0

AP7

AP6

BP7

BP6

AP3

AP2

BP3

BP2

CP1

CP0

CP3

CP2

CP1

CP0

CP3

CP2


Logs vs. Metrics

Logging data killed the metrics cluster


Quality of Service with Kafka

Brokers

Consumers

Producers

AP1

AP0

BP1

BP0

AP5

AP4

BP5

BP4

AP3

AP2

BP3

BP2

AP7

AP6

BP7

BP6

AP5

AP4

BP5

BP4

AP1

AP0

BP1

BP0

AP7

AP6

BP7

BP6

AP3

AP2

BP3

BP2

CP1

CP0

CP3

CP2

CP1

CP0

CP3

CP2


Deployment Nightmares

Parallel deployment wasn’t possible so…

Babysitting sequential deployments


Easy deployments

Kafka 0.8.1 makes sure the cluster is in a good state before shutting down

– If any brokers in the cluster have under replicated partitions, Kafka will not shut down

– Kafka ensures that only 1 broker is in shutdown sequence at a time.


Killing Zookeeper

Consumer offset management done within Zookeeper

Every consumer committing offsets every minute for every partition makes ZK very unhappy.


Zookeeper on SSD


Monitoring


Kafka Is Broken!


Kafka Is Broken!

Everything is Kafka’s fault first

What is lag?

Consumer Problems– Application problems– Kafka client problems


How Do We Sleep At Night?

Educating Users– Why lag is their fault

Monitoring the Ecosystem– Kafka Brokers– Zookeeper– Mirror Makers– Audit– REST Interfaces

Week Over Week


Cluster Health and Utilization

Under replicated partitions

Offline partitions

Broker partition count

Data size on disk

Leader partition count

Network utilization


Zookeeper

Ensemble availability

Latency

Outstanding requests


Mirror Maker and Audit

Mirror Maker– Lag– Dropped Messages

Audit Consumer– Lag– Completeness check

Audit UI

Producer

Cluster ClusterMM

MessagesMessageCounts

AuditConsumer

AllMessages

AuditState

AuditConsumer

AuditUI

AuditState


Audit UI


Tuning


Hardware and OS

Kernel Tuning– Swapping is Death– Allow more dirty pages– Allow less dirty cache

Disk throughput– More spindles– Longer commit interval


Java Virtual Machine


Garbage Collection


Garbage Collection

Java 7, update 51

Garbage First (G1) Collector– Set the heap size– Specify a target GC pause time– Don’t set the New size

GC Times– Less than 15ms per second in GC– Steady 20-22ms GC intervals– Almost no full GC cycles (and only 200-400ms when it does)


Closing


What’s Coming in 0.8.2

Consumer offsets in the broker

Delete topic

Further down the road– New producer– Improved producer API


Upcoming Operational Work

Learning to share

Shrinking a cluster

Cluster comparison

Advanced monitoring


How Can You Get Involved?

http://kafka.apache.org

Join the mailing lists– [email protected]

irc.freenode.net - #apache-kafka

Contribute tools

http://kafka.apache.org/

mailto:[email protected]



Talk To Us

Kafka SREs at LinkedIn– Clark Haskins

https://www.linkedin.com/in/clarkhaskins [email protected]

– Todd Palino https://www.linkedin.com/in/toddpalino [email protected]

https://www.linkedin.com/in/clarkhaskins

https://www.linkedin.com/in/clarkhaskins


https://www.linkedin.com/in/toddpalino

https://www.linkedin.com/in/toddpalino



Questions

Enterprise Kafka: Kafka as a Service

Data & Analytics

Transcript of Enterprise Kafka: Kafka as a Service

Kafka to the Maxka - (Kafka Performance Tuning)

Challenges of a multi tenant kafka service

Paris Kafka Meetup - How to develop with Kafka

Kafka Connect & Streams - the ecosystem around Kafka

Kafka Low-Level Design discussion of Kafka Design Kafka …cloudurable.com/ppt/4-kafka-detailed-architecture.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting

Kafka Streams: Hands-on Session - ce.uniroma2.it · Kafka Streams Kafka Streams: • Kafka Streams is a client library for processing and analyzing data stored in Kafka • Supports

Enterprise Sales Engineer - Sumo Logic James …...Deploying Apache Kafka Exabyte Scale Data Coordination James Mountifield Enterprise Sales Engineer - Sumo Logic Sumo Logic Confidential

Publish-subscribe Message Framework with Apache Kafka and ...vvtesh.co.in/teaching/bigdata-2020/slides/studentppt/kafka-kinesis.pdf · Introduction of Kafka Kafka elementary concepts

Cloudurablecloudurable.com/ppt/cloudurable-kafka-intro-with-simple-java-produc… · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting ™ Kafka Fundamentals

Formatted: Figure [PACKT] cm, Width: 21.59 cm, Height: 27 ... · Kafka 0.7.x Consumer Kafka 0.7.x Cluster Kafka Migration Kafka 0.8 Cluster Kafka 0.8 Producer Producer (Front End)

PDF: Kafka Tutorial - Cloudurablecloudurable.com/ppt/cloudurable-kafka-tutorial-v1.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting ™ What is Kafka?

Cassandra and Kafka Support on AWS/EC2cloudurable.com/ppt/kafka-tutorial-cloudruable-v2.pdf · Cassandra / Kafka Support in EC2/AWS. Kafka Training, Kafka Consulting Cassandra and

MAURICE BLANCHOT, de Kafka a Kafka

Forcepoint Behavioral Analytics Installation Manual › content › support › library › ... · Kafka kafka 9092-9095 API, Rose Kafka Manager kafka 9000 Administrator Workstation

Administering Oracle Event Hub Cloud Service Dedicated · Configuring Kafka Client with Oracle Event Hub Cloud Service — Dedicated CA 5-1 Configuring Kafka Brokers and Clients with

Kafka Tutorial - basics of the Kafka streaming platform

· Apache Kafka Introduction to Apache Kafka Apache Kafka Architecture explanation Practical Examples on Apache Kafka SCALA, PYTHON, SPARK Course Content

Enterprise Kafka: Kafka as a Service

Self-Service Data Ingestion Using NiFi, StreamSets & Kafka

Tungsten Replicator for Kafka, Elasticsearch, Cassandra to Kafka+Elastic... · •Full transaction support for Kafka •Support for Amazon Elasticsearch •Kafka Extraction –Parsing