Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of...
Transcript of Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of...
![Page 1: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/1.jpg)
Apache Kafka and the Rise of Event-Driven Microservices
Jun Rao Co-founder of Confluent
![Page 2: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/2.jpg)
LinkedIn at 2010 : World’s Largest Professional Network
Members Worldwide
2 new Members Per Second
100M+ Monthly Unique Visitors
200M+ 2M+ Company Pages
Connecting Talent Opportunity. At scale…
2
![Page 3: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/3.jpg)
It’s all about data!
3
Value ↑
Insights ↑
Product
Science Data
User
Virality ↑
Signals ↑
![Page 4: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/4.jpg)
Initial database driven architecture
database
web application
web application
![Page 5: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/5.jpg)
Realization #1: Event > State
• State: I work at Confluent • Event: I changed job to work at Confluent
![Page 6: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/6.jpg)
Event driven microservices
member recommendation
search index
graph engine
new job description
![Page 7: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/7.jpg)
Realization #2: leverage non-transactional data
• Business metrics – clicks, search keywords, pageviews
• Operational metrics – requests/sec, request types/sec
• Application logs – service calls, errors
• IOT • …
![Page 8: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/8.jpg)
Database a mismatch for both!
![Page 9: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/9.jpg)
Mismatch #1: no first class API for events
database
log
table
member recommendation
search index
graph engine
SQL
SQL
SQL
Tremendous load pressure on database!
![Page 10: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/10.jpg)
Mismatch #2: not suitable for non-transactional data
• 1000X more volume • Different transactional needs • Not always needing a relation view
![Page 11: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/11.jpg)
Danger of Point-to-point Pipelines
![Page 12: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/12.jpg)
Ideal Architecture
![Page 13: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/13.jpg)
1st Attempt: Don’t Reinvent the Wheels
• Why not messaging systems?
![Page 14: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/14.jpg)
![Page 15: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/15.jpg)
Version 1 of Kafka
• High throughput pub/sub – Design 1: make log first class citizen – Design 2: distributed architecture
![Page 16: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/16.jpg)
Design #1: log as first a class citizen
16
database
log
table
long poll() API
![Page 17: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/17.jpg)
Design #1: log as first a class citizen
17
database
log
table
long poll() API
Easy to optimize for throughput
![Page 18: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/18.jpg)
Design #1: log as first a class citizen
18
database
log
table
long poll() API
Persistency for lagging/rewinding consumption
![Page 19: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/19.jpg)
Design #1: log as first a class citizen
19
database
log
table
long poll() API
Ordered delivery to reduce consumer bookkeeping overhead
![Page 20: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/20.jpg)
Design #2: distributed architecture
20
topicA-0
topicB-0
topicC-0
broker 1
topicA-1
topicB-1
topicC-1
broker 2 topicA-2
topicB-2
topicC-2
broker 3
topicA-3
topicB-3
topicC-3
broker 4
Kafka cluster
producer producer producer
consumer consumer consumer
![Page 21: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/21.jpg)
Kafka at LinkedIn in 2011
• 28 billion messages/day • 460 thousand messages written/sec • 2.3 million messages read/sec • Tens of thousands of producers
– Every production service is a producer
• Data democracy!
![Page 22: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/22.jpg)
Kafka => Apache in 2011
6 of the top 10 travel companies
8 of the top 10 insurance companies
7 of the top 10 global banks
9 of the top 10 telecom companies
![Page 23: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/23.jpg)
Royal Bank of Canada Event-Driven Banking
30+ Use-cases
50+ apps
10+ different lines of businesses
Lowering anomaly detection from weeks to real-time
Digital Marketing Security
Consumer Credit Services
SaaS
Corporate Real Estate
Investor Services
Treasury Services
….
Fraud Data Warehouse
Microservices
![Page 24: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/24.jpg)
Carnival cruise line
![Page 25: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/25.jpg)
Building the processing layer
event-driven microservice
Kafka pub/sub
event-driven microservice
event-driven microservice
• Transformation • Enrichment • Aggregation
![Page 26: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/26.jpg)
Kafka Streams
KStream<Integer, Integer> input = builder.stream(“numbers-topic”); // Stateless computation KStream<Integer, Integer> doubled = input.mapValues(v -> v * 2); // Stateful computation KStream<Integer, Integer> sumOfOdds = input .filter((k,v) -> v % 2 != 0) .selectKey((k, v) -> 1) .reduceByKey((v1, v2) -> v1 + v2, ”sum-of-odds") .toStream();
![Page 27: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/27.jpg)
KSQL (from Confluent)
CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
![Page 28: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/28.jpg)
Event driven platform
database
event-driven microservice kstreams/ksql transactional
events
non-transactional events
Kafka pub/sub
event-driven microservice
event-driven microservice
kstreams/ksql
kstreams/ksql
![Page 29: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/29.jpg)
Still interesting work ahead
• Scalability in metadata • Streaming database • Cloud integration
![Page 30: Apache Kafka and the Rise of Event-Driven Microservices · Apache Kafka and the Rise of Event-Driven Microservices Jun Rao Co-founder of Confluent](https://reader030.fdocuments.us/reader030/viewer/2022040813/5e581dd68caba51005692e9b/html5/thumbnails/30.jpg)
Conclusion
• The success for business not only depends on software, but how they build software
• Apache Kafka offers a new platform than traditional database
• This is an exciting time to work on streams