Large Message Problem #

Apache Kafka is deeply designed and optimized to handle massive small-sized stream data traffic — typically ranging from a few hundred bytes to a few tens of kilobytes per message (like log messages, click events, or financial transactions). However, in real-world implementations, we’re often faced with the need to send large files (for example, scanned PDF documents, high-resolution images, audio recordings, or zip archives above 1 MB). Forcing these large files directly into Kafka without a special strategy is one of the worst architectural mistakes (anti-patterns). This not only triggers instant delivery failures, but can also degrade the stability and speed of the entire broker cluster. Therefore, we must understand the physical consequences of large messages, how to strictly align parameters if forced to do so, and how to implement the Claim Check Pattern architecture solution as the best way out.

Why Sending Large Files Is an Anti-Pattern in Kafka #

Many developers assume Kafka is like a file storage system or FTP server that can hold binary payloads of any size. This assumption is wrong because Kafka operates at very strict memory and disk I/O levels.

Here are the bad impacts of sending large messages directly to Kafka:

1. Head-of-Line Blocking Problems #

Kafka brokers serve partition reads/writes using one I/O thread per socket. If a producer sends one 10 MB message to a topic partition, the broker spends several seconds reading that binary packet from the network socket and writing it to disk. During this large file write process, the broker blocks the queue of hundreds of thousands of other small messages waiting behind that large message. Cluster latency spikes dramatically.

2. Heap Memory Pressure and Garbage Collection Overhead #

When a broker receives a large message, it must allocate JVM heap memory space to hold that message before writing it to disk. If several producers send large messages in parallel, broker heap usage spikes suddenly. This condition forces the Java Garbage Collector to run full memory cleanup cycles (Full GC pause). During this GC pause, the broker freezes, stops responding to cluster heartbeats, gets suspected dead by the controller, and triggers unnecessary leader elections that disrupt the cluster.

3. Expensive Disk Waste #

Broker disks are usually fast storage media (SSD/NVMe) with expensive RAID configurations to support high I/O speeds. Storing gigabyte-capacity images or PDF documents in Kafka binary disks is very cost-inefficient, especially since that data gets replicated by our Replication Factor (e.g., multiplied 3 times across brokers).

Payload Size Parameter Alignment (The Golden Triangle) #

By default, Kafka limits the maximum message size to 1 MB (1,048,576 bytes). If our payload is slightly above 1 MB (for example, 1.5 MB) and we absolutely must send it directly to Kafka, we must raise the size limit parameters.

This tuning must not be done carelessly. We must align three parameters on the producer, broker, and consumer sides to avoid data transmission failures. This alignment rule is often called The Golden Triangle:

$$\text{max.request.size (Producer)} \le \text{message.max.bytes (Broker)} \le \text{max.partition.fetch.bytes (Consumer)}$$

1. Producer-Side Configuration: max.request.size #

Determines the maximum size limit of one Produce Request (write request) that a producer may send to the broker. If our ProducerRecord object exceeds this limit, the producer client immediately throws the RecordTooLargeException locally before the data is sent to the socket.

2. Broker-Side Configuration: message.max.bytes #

Determines the maximum binary message size allowed to be written to a broker partition log. This parameter can be set globally in the server.properties file or per topic (through the max.message.bytes topic property). If a producer sends data above this limit, the broker immediately rejects it and returns the RecordTooLargeException error.

3. Consumer-Side Configuration: max.partition.fetch.bytes #

Determines the maximum memory size limit per partition that a consumer fetches in one data fetch request. This parameter must be set greater than or equal to the broker’s message.max.bytes value. Otherwise, our consumer can never read those large messages from the broker and the consumption process stops forever (stuck) because the consumer lacks memory capacity to deserialize those messages.

[!CAUTION] We must also adjust the consumer property fetch.max.bytes (default: 52,428,800 or 50 MB) to limit the total combined message size from all partitions read in one fetch so it doesn’t trigger OutOfMemory on our consumer application side.

Modern Architecture Solution: Claim Check Pattern #

To avoid all the problems above, modern microservices architectures apply the Claim Check Pattern (also known as Reference-Based Messaging). This pattern separates heavy binary payloads from Kafka’s lightweight event message path.

How the Claim Check Pattern Works: #

  1. Upload Payload: The producer application thread detects a large file. Instead of sending it to Kafka, the producer uploads the file to an external Cloud Object Storage system (like AWS S3, Google Cloud Storage, or local MinIO).
  2. Get Pointer: Object storage processes the upload and returns a unique pointer URL referencing that file (for example: s3://bucket-transaksi/dokumen-12345.pdf).
  3. Send Lightweight Metadata: The producer assembles a lightweight JSON metadata message containing that pointer URL, then sends it to the Kafka topic. This message is only a few hundred bytes.
  4. Read Metadata: Consumers read the metadata message from the Kafka topic quickly without burdening the broker disk.
  5. Download Payload: The consumer extracts the pointer URL from the metadata message, then asynchronously downloads the physical binary file directly from AWS S3 using the relevant object storage client SDK.
flowchart TD
    subgraph Direct["Approach 1: Direct Write (Anti-Pattern)"]
        direction TB
        App1["Application Thread (10MB Large File)"] -->|"Send Physical Payload"| Prod1["Kafka Producer Client"]
        Prod1 -->|"ProduceRequest (10MB)"| Broker1["Kafka Broker (orders topic)"]
        Broker1 -->|"Writing 10MB to Disk (Very Slow)"| Disk1["Broker Disk"]
    end
    
    subgraph ClaimCheck["Approach 2: Claim Check Pattern (Recommended)"]
        direction TB
        App2["Application Thread (10MB Large File)"] -->|"1. Upload Physical Payload"| S3["Object Storage (AWS S3 / GCS)"]
        S3 -->|"2. Return Pointer URL (e.g., s3://bucket/file-uuid)"| App2
        App2 -->|"3. Send Lightweight Metadata (JSON)"| Prod2["Kafka Producer Client"]
        Prod2 -->|"4. ProduceRequest (1 KB)"| Broker2["Kafka Broker (orders topic)"]
        Broker2 -->|"Write 1 KB to Disk (Instant)"| Disk2["Broker Disk"]
    end
    
    style Direct stroke:#e5e7eb
    style ClaimCheck stroke:#e5e7eb
    style S3 stroke:#2e7d32,stroke-width:2px
    style Disk1 stroke:#c62828,stroke-width:2px
    style Disk2 stroke:#2e7d32,stroke-width:2px

Claim Check Pattern Advantages: #

  • Kafka Performance Stays Stable: The broker only serves ultra-fast metadata messages, minimizing latency and GC overhead.
  • Unlimited Scalability: AWS S3 or Google Cloud Storage is specifically designed to store large files massively at costs far cheaper than broker SSDs.
  • Failure Isolation: Network failures downloading large files on the consumer side don’t disrupt the message flow in other Kafka partitions.

Claim Check Pattern Implementation in Java #

Here’s a Java code example comparing raw large binary file delivery handling (anti-pattern) versus the safe Claim Check Pattern implementation using mocked AWS S3 storage:

// ANTI-PATTERN: Sending large raw binary PDF files directly to Kafka
// Triggers broker rejection or blocks other message queues
public class NaiveImageProducer {
    public void sendLargeFile(KafkaProducer<String, byte[]> producer, String orderId, byte[] rawPdfBytes) {
        // ✗ The 15MB rawPdfBytes payload is sent directly
        ProducerRecord<String, byte[]> record = new ProducerRecord<>("orders-pdf-topic", orderId, rawPdfBytes);
        producer.send(record); // Triggers RecordTooLargeException!
    }
}

// CORRECT: Using the Claim Check Pattern (Reference-Based Messaging)
// Storing the physical file in S3, and only sending the pointer URL in Kafka
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ObjectMetadata;
import java.io.ByteArrayInputStream;
import java.util.UUID;

public class ClaimCheckOrderProducer {
    private final AmazonS3 s3Client;
    private final String s3BucketName = "my-large-orders-bucket";

    public ClaimCheckOrderProducer(AmazonS3 s3Client) {
        this.s3Client = s3Client;
    }

    public void sendOrderWithLargeFile(KafkaProducer<String, String> kafkaProducer, String orderId, byte[] rawPdfBytes) {
        // 1. Create a unique UUID for the S3 file name
        String fileKey = "orders/" + orderId + "-" + UUID.randomUUID().toString() + ".pdf";

        // 2. Upload the physical binary file directly to AWS S3
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentLength(rawPdfBytes.length);
        metadata.setContentType("application/pdf");

        System.out.printf("Uploading file %s to S3 (%d bytes)...%n", fileKey, rawPdfBytes.length);
        s3Client.putObject(s3BucketName, fileKey, new ByteArrayInputStream(rawPdfBytes), metadata);

        // 3. Get the pointer reference URL (Claim Check)
        String s3PointerUrl = String.format("s3://%s/%s", s3BucketName, fileKey);

        // 4. Create lightweight JSON metadata to send to Kafka
        String kafkaJsonPayload = String.format(
            "{\"orderId\":\"%s\",\"claimCheckUrl\":\"%s\",\"sizeBytes\":%d}",
            orderId, s3PointerUrl, rawPdfBytes.length
        );

        // 5. Send the lightweight metadata (1 KB) to Kafka
        // ✓ CORRECT: The broker disk I/O load is very light, cluster performance stays stable
        ProducerRecord<String, String> record = new ProducerRecord<>("orders-metadata-topic", orderId, kafkaJsonPayload);
        kafkaProducer.send(record, (recMetadata, exception) -> {
            if (exception != null) {
                System.err.println("Failed to send metadata to Kafka: " + exception.getMessage());
            } else {
                System.out.printf("Metadata successfully sent to partition %d, offset %d%n", 
                    recMetadata.partition(), recMetadata.offset());
            }
        });
    }
}

Security and Data Cleanup in the Claim Check Pattern #

When implementing the Claim Check Pattern, there are two non-functional operational aspects we must seriously consider in production: Data Access Security and File Cleanup Lifecycle.

1. Securing Payload Access Using Presigned URLs #

Sending a static URL pointing directly to sensitive files in object storage (like s3://my-bucket/orders/pdf-1.pdf) can trigger security holes. Every consumer service reading those Kafka messages can access the file indefinitely.

  • Security Solution: Producers can be configured to generate Presigned URLs with short expiration limits (for example, 15 minutes). Consumers must immediately download the file before the expiration passes. This guarantees that even if Kafka event data leaks or is read by others later, the physical download URL can no longer be reused.

Here’s a Presigned URL generator example using the AWS SDK for Java:

// ✓ CORRECT: Using the AWS SDK to generate Presigned URLs with short expiration limits
import com.amazonaws.HttpMethod;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import java.net.URL;
import java.util.Date;

public class SecurePointerGenerator {
    public String generatePresignedUrl(AmazonS3 s3Client, String bucketName, String fileKey) {
        Date expiration = new Date();
        long expTimeMillis = expiration.getTime() + 1000 * 60 * 15; // URL valid for 15 minutes
        expiration.setTime(expTimeMillis);
        
        GeneratePresignedUrlRequest request = new GeneratePresignedUrlRequest(bucketName, fileKey)
            .withMethod(HttpMethod.GET)
            .withExpiration(expiration);
        
        URL url = s3Client.generatePresignedUrl(request);
        return url.toString(); // Returns a unique token URL
    }
}

2. Automating File Cleanup in Object Storage #

Unlike Kafka brokers which have automatic time-based log deletion policies (retention.ms), external object storage systems (AWS S3/MinIO) don’t automatically delete our uploaded files just because the related Kafka messages have expired. Without handling, S3 data sizes keep growing unboundedly and accumulate operational costs.

  • Cleanup Solution: We must configure an S3 Lifecycle Policy on the related storage bucket. We can create automatic rules in AWS S3 to detect files under the orders/ prefix and permanently delete them after 7 or 14 days, aligned with our Kafka topic retention parameters.

Claim Check Pattern Implementation on the Consumer Side (Downloader) #

On the consumer side, our application must deserialize the lightweight JSON metadata from Kafka, extract the pointer URL, then use the object storage client SDK to download the original physical payload.

Here’s a Java consumer implementation example:

// CORRECT: The consumer reads lightweight metadata, then downloads the physical binary file from S3 separately
import org.apache.kafka.clients.consumer.ConsumerRecord;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.S3Object;
import java.io.InputStream;

public class ClaimCheckConsumer {
    private final AmazonS3 s3Client;
    private final ObjectMapper objectMapper = new ObjectMapper();

    public ClaimCheckConsumer(AmazonS3 s3Client) {
        this.s3Client = s3Client;
    }

    public void processRecord(ConsumerRecord<String, String> record) {
        try {
            // 1. Deserialize the lightweight JSON metadata from Kafka
            OrderMetadata metadata = objectMapper.readValue(record.value(), OrderMetadata.class);
            String s3Url = metadata.getClaimCheckUrl(); // Example: "s3://my-large-orders-bucket/orders/pdf-1.pdf"
            
            // 2. Extract the bucket name and key from the reference URL
            String bucketName = s3Url.substring(5, s3Url.indexOf("/", 5));
            String fileKey = s3Url.substring(s3Url.indexOf("/", 5) + 1);
            
            // 3. Download the physical file separately from S3
            System.out.printf("Downloading payload from S3: %s...%n", fileKey);
            S3Object s3Object = s3Client.getObject(bucketName, fileKey);
            
            try (InputStream in = s3Object.getObjectContent()) {
                byte[] fileBytes = in.readAllBytes();
                // ✓ Process the binary PDF file safely at the application level
                processPdf(metadata.getOrderId(), fileBytes);
            }
        } catch (Exception e) {
            System.err.println("Failed to process claim check record: " + e.getMessage());
        }
    }

    private void processPdf(String orderId, byte[] pdfBytes) {
        System.out.printf("Successfully processed the PDF file for Order ID %s%n", orderId);
    }
}

Another Alternative: Message Splitting (Chunking) #

If we don’t have access to external Cloud Object Storage, the second alternative for handling large files is Chunking (Message Splitting).

Chunking Concept #

  • Producer Side: A 10 MB file is split into 10 small parts of 1 MB each in the producer JVM memory. Each part is wrapped with special metadata:
    • File_UUID: The unique ID of the whole file.
    • Chunk_Index: The chunk index (0 to 9).
    • Total_Chunks: The total chunk count (10).
    • Each chunk is sent as an independent message to Kafka.
  • Consumer Side: Consumers must have a local assembly buffer memory area. The consumer reads those chunks, holds them in memory, reassembles them by Chunk_Index, and executes the file after all 10 parts are complete.

Fatal Chunking Weaknesses: #

  1. Ordering Loss Problems: If chunks are sent to different partitions (or their order changes due to network retry failures without idempotence), reassembling the file on the consumer is very complex and prone to corruption.
  2. Consumer Memory Leaks: Consumers must allocate large RAM to hold incomplete file chunks from thousands of different senders. If one chunk is lost in transit, the consumer holds the remaining chunks in memory forever, triggering Memory Leaks.
  3. Code Complexity: Burdens developer teams to write highly bug-prone file assembly state management code.

Therefore, the Claim Check Pattern remains the safest industry standard for handling large message problems.

Summary #

  • Large Message Danger: Sending messages above 1 MB directly triggers head-of-line blocking problems and GC overhead that can cripple the cluster.
  • The Golden Triangle: Absolute alignment between the max.request.size (producer), message.max.bytes (broker), and max.partition.fetch.bytes (consumer) properties.
  • Claim Check Pattern: The strategy of storing large files in external object storage (AWS S3) and only sending lightweight reference pointer URLs to Kafka.
  • Cost Efficiency: Avoids expensive binary storage on Kafka broker disks that get replicated repeatedly.
  • Chunking Alternative: The technique of splitting large files into small parts sent separately, but carries consumer memory leak risks.
  • Metadata Separation: Keeps the Kafka message transmission path lean and fast, only for lightweight structured event data.

← Previous: Over-Partitioning Next: Producer Misconfiguration →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact