Introduction to Kafka with Spring Integration

Introduction to Kafka with Spring Integration

• Kafka (Mihail Yordanov)

• Spring integration (Borislav Markov)

• Students Example (Mihail & Borislav)

• Conclusion

Motivation

• Real time data being continuously generated

• Producers and consumers relationships

• Streaming systems

• Messaging systems

Apache Kafka• Developed by LinkedIn• “We built Apache Kafka at LinkedIn with a specific

purpose in mind: to serve as a central repository of data streams.” - Jay Kreps

• Kafka improvements– Transporting data between systems– Richer analytical processing

• “A publish-subscribe messaging system rethought as a distributed commit log”

Kafka Vocabulary

• Brokers

• Producers

• Consumers

• “A publish-subscribe messaging system rethought as a distributed commit log”

Kafka Vocabulary

• Topics

• Partitions

• “A publish-subscribe messaging system rethought as a distributed commit log”

Kafka Speed

Kafka Speed

Kafka Speed

Consumer groups

• Definition - “A publish-subscribe messaging system rethought as a distributed commit log”

• Publish - Subscribe

• Queueing

Spring Integration

• Spring Integration enables lightweight messaging

• Supports integration with external systems via declarative adapters

• Primary goal is to provide a simple model for building enterprise integration solutions

• Separation of concerns • Maintainable, testable code

Features• Implementation of most of the Enterprise Integration Patterns

• Endpoint

• Channel (Point-to-point and Publish/Subscribe)

• Aggregator

• Filter

• Transformer

• Control Bus

• …

• Integration with External Systems

• ReST/HTTP

• ...

The theory...

The patterns can be found athttp://www.enterpriseintegrationpatterns.com/patterns/messag

ing/index.html

Message• Messages are objects that carry information between two applications

• They are constructed by the producer and they are consumed/deconstructed at

the consumer/subscriber

• Message consists of:

– payload

– header

public interface Message<T> {

T getPayload();

MessageHeaders getHeaders();

}

Message endpointsCommon endpoints:• Service Activator - invokes a method on a

bean• Message Bridge - couples different

messaging modes/adapters. Example: (P2P with Publish/Subscribe)

• Message Enricher - enrich the incoming message with additional information.– Header Enricher - add header

attributes– Payload Enricher - enrich payload

Message endpoints (continued)• Gateway - Used when we don’t have knowledge for the

messaging system. – Synchronous Gateway - void or returns T– Asynchronous Gateway - returns Future<T>

• Delayer - introduce delay between sender and receiver

• Consumers

– Polling Consumers - poll messages in a timely fashion

– Polling Using Triggers - poll messages with PeriodicTrigger

and with CronTrigger

– Event-Driven Consumers - they wait for someone to

deliver the message (framework’s responsibility)

Message endpoints (continued)

• Channel Adapter - endpoint that connects a Message Channel to some other system or transport.

– inbound

–outbound

Message Channels• interface MessageChannel

– boolean send(Message<?> message);

– boolean send(Message<?> message, long timeout);

• Point-to-Point Mode -> PollableChannel

– Message<?> receive();

– Message<?> receive(long timeout);

• Publish/Subscribe Mode -> SubscribableChannel

– boolean subscribe(MessageHandler handler);

– boolean unsubscribe(MessageHandler handler);

Message Channels(continued)

•Queue Channel - has capacity•Priority Channel - queue channel with

prioritization•PublishSubscribe Channel•DirectChannel - mixed type P2P and

Pub/Sub

Message Channels(continued)

Flow components• Filters

– Custom filters are tied to the framework, can be bean method returning boolean

– Framework MessageSelector - use of method boolean accept(Message m)

– Using annotation - @Filter • Routers

– PayloadTypeRouter– HeaderValueRouter– Custom Routers - any bean method can be

router, the result will be the channel name. – Recipient List Router - the message goes to all

statically defined channels

Flow components (continued...)• Splitters - splits a message into pieces

– Custom splitters - any bean method;– Framework AbstractMessageSplitter - use of method Object splitMessage(Message

m);– Using annotation - @Splitter ;

• Aggregator– assemble multiple messages to create a single parent message– Complex task - all messages of a set have to arrive before aggregators can start work– CorrelationStrategy - defines the key for grouping of the messages, the default

grouping is based on CORRELATION_ID – ReleaseStrategy - dictates at which point the group of messages should be sent or

released for aggregation. Default is SequenceSizeReleazeStrategy– Message Store - aggregators hold messages until all of them arrived. Options are:

• in-memory• external database

• Resequencer - fix order of the messages(work on SEQUENCE_NUMBER)

Transformers• Built-in transformers

– String Transformers - invokes toString() method of the payload

– Map Transformers - transformes POJO to a name-value pair of

Map

– Serializing and Deserializing Transformers - converts payload

to byte array

– JSON Transformers - object-to-json converts POJO to readable

JSON format; json-to-object transformer needs additional

property “type”

– XML Transformers - requires Spring OXM (Object-to-XML);

org.springframework.oxm.Marshaller/Unmarshaller

Transformers(continued...)

• Built-in transformers

– XPath Transformers - decodes

XML using XPath expressions, ex.:

xpath-

expression=”/mytag/@prop”

• Custom Transformers - can be any

spring bean method

• Using @Transformer

Students Example

• Example application will demo usage

of Kafka and Spring Integration

• App is built with maven

• Ideal candidate for Microservice

• Idea: takes students from outside and

calculates their average score

Maven dependencies

Project Structure

• Package “api” – POJO models

• Package “config” – beans with factory methods

• Package “service” – the business logic

• Folder “resources” –application.properties

Spring Integration DSL

Students Example

Service activator

Running the app and giving some inputs

Check the logs

Conclusion

Why Kafka and Spring ?• Easy for microservices

• Clean and testable code

• Widely adopted technologies

• Easy to be dockerized