Claim check
Large messages in Kafka
Claim check is an open source Kafka Interceptors library that handles large messages by implementing the Claim check pattern.
Message size limitations
The default Kafka broker configuration specifies a default limit around 1Mb for message batches. In some use cases you might face messages that vary in size and might exceed the limit. It seems to be theoretically possible to increase this limit in your cluster, but it involves changing default settings both between client/broker and for broker/broker replication.
The Claim check integration pattern
The Claim Check Kafka interseptors library implements the Claim Check integration pattern for Kafka Java clients.
The analogy is here from the world of airline travel; the Sender (passenger) wants to send a large message (transport heavy and bulky luggage) over a message bus (airplane). Instead of sending your message over the normal message bus (carry-on luggage), you store your message in a separate datastore (check in your luggage for transport in the storage compartment). The consumer (passenger at the destination) fetches the message form the separate datastore (retrieves the luggage from the carousel).
Solution Design
The Claim Check Kafka interseptors library stores the large message in Azure Blob Storage, but any key-value store is possible.
The library is implemented using the concepts of (De-)Serializers and Interceptors in the standard Kafka Java client. This means that any existing software that uses the Java Kafka client and allows you to add config settings dynamically to the Kafka Producer and Consumer can keep working by just adding the Claim Check Interceptors library as well as a dependency to the key-value storage of your choice.
Information
Version:
No of downloads: 500+
Supported backends: Azure Blob Storage (tell us what you want next)