Producer #

In the Apache Kafka ecosystem, a Producer is a client application that publishes or writes event streams into Kafka cluster topics. Although the task sounds simple — sending messages to a server — the internal workings of the Kafka producer SDK are actually very complex and sophisticated. The producer adopts an asynchronous architecture highly optimized for high performance, smart compression, and zero data loss delivery guarantees. Understanding the producer’s internal workflow and its critical configuration parameters is essential to avoid memory leaks, duplicate data, or losing important data at the application level.

Internal Producer Architecture: How Are Messages Sent? #

When we call the send() function in a producer application, the message isn’t sent directly to the network and received by the broker server. Instead, the message must pass through a series of very strict internal processing stages in our application’s local memory before finally being sent over the network by a background thread.

Let’s study the data journey inside the Kafka Producer SDK through the following diagram:

flowchart TD
    subgraph SDKProducer["Producer SDK Internal Cycle"]
        direction TB
        Input["New Message (Key, Value)"] --> Serializer["1. Serializer <br/> (Convert Object to Binary/Byte Array)"]
        Serializer --> Partitioner["2. Partitioner Engine <br/> (Determine Target Physical Partition)"]
        Partitioner --> Accumulator["3. Record Accumulator <br/> (Hold Message Batches in Memory)"]
    end

    subgraph NetworkIO["Network Delivery"]
        SenderThread["4. Sender Thread <br/> (Background I/O Thread)"]
    end

    subgraph BrokerCluster["Kafka Cluster"]
        BrokerLeader[("Leader Broker <br/> (Physical Storage)")]
    end

    Accumulator -->|Batch Full / Timeout| SenderThread
    SenderThread -->|Send Binary Packet| BrokerLeader

Let’s break down the important role of each component above in depth:

1. Serializer #

The first stage is Serialization. Because Kafka brokers only understand data as raw byte arrays, producers must translate data objects from our programming language (like Java objects, Python Map types, or Go Structs) into a binary representation.

The Kafka SDK provides built-in serializers for common data types like StringSerializer, IntegerSerializer, and ByteArraySerializer. However, for complex business data, we must use specialized serializers integrated with a Schema Registry, such as Apache Avro or Protobuf serializers.

2. Partitioner #

After the data is converted to binary, the Partitioner component determines which partition the message should be routed to. If we include a Key, the Partitioner uses the Murmur2 hash modulo formula. If the key is null, the Partitioner uses the Sticky Partitioning algorithm to group messages into the same partition to improve network efficiency.

3. Record Accumulator #

This is where Kafka’s main efficiency lies. Messages whose partition has been determined aren’t sent directly; they’re placed into the Record Accumulator. This is a producer-local memory area that groups messages into several batch queues based on destination partition. Each partition has its own batch queue. Messages keep piling up in this buffer memory until they meet the delivery criteria.

4. Sender Thread #

The Sender Thread is a dedicated background I/O thread that continuously monitors the Record Accumulator. Its job is to take ready message batches from the accumulator, turn them into TCP socket requests, and send them in parallel to the Kafka broker acting as the Leader of each destination partition.

Batching and Latency Mechanisms (batch.size & linger.ms) #

To achieve maximum data throughput, Kafka producers rely heavily on batching. Sending 10,000 messages individually one by one over the network triggers huge TCP protocol overhead and stalls servers. Sending 10,000 messages at once in one large batch is far more efficient.

We can control the balance between data processing speed (throughput) and delivery delay (latency) using two main configuration parameters:

1. batch.size (Maximum Batch Size) #

This parameter sets the maximum memory limit in bytes for one message batch per partition. The default value is 16384 bytes (16 KB). If the producer generates data so fast that message accumulation for one particular partition reaches 16 KB, the producer immediately closes that batch and hands it to the Sender Thread for immediate delivery, even if the wait time hasn’t elapsed.

If we have enough server memory and very high data throughput, raising this value to 64 KB or 128 KB is highly recommended to improve data compression efficiency.

2. linger.ms (Maximum Wait Time) #

This parameter determines how long the producer delays message delivery in buffer memory to give other messages a chance to join the same batch. The default value is 0 milliseconds (meaning messages are sent immediately without waiting for a full batch).

By raising the linger.ms value (for example, to 20 milliseconds), we instruct the producer to wait up to 20 ms before sending the batch. This adds a slight artificial latency (20 ms), but greatly increases system throughput because far more messages are sent in one network packet and data compression works much more optimally.

The Acknowledgment (acks) Concept and Data Reliability #

Data safety is a top priority in distributed messaging system architecture. Apache Kafka allows producers to choose the desired data delivery guarantee level through the acks (Acknowledgments) configuration parameter.

The acks value determines how many partition replicas on the broker side must confirm data receipt before the broker sends a success signal back to the producer:

flowchart LR
    subgraph AcksSetting["Acks Configuration Options"]
        Acks0["acks=0 <br/> (No Confirmation)"]
        Acks1["acks=1 <br/> (Leader Only Confirms)"]
        AcksAll["acks=all / -1 <br/> (Leader & All ISR Confirm)"]
    end

1. acks = 0 (Maximum Speed, High Loss Risk) #

The producer sends messages to the broker and immediately assumes delivery succeeded without waiting for any confirmation or reply from the broker server.

  • Analogy: Sending regular mail via post without tracking.
  • Advantage: Very low latency and maximum throughput because there’s no network wait time for replies.
  • Disadvantage: Very high data loss risk. If the broker crashes right before writing the message to disk, the data is lost forever without the producer knowing. Only suitable for non-critical metric data or clickstream tracking logs.

2. acks = 1 (Moderate Balance) #

The producer waits for a success confirmation from the Leader broker of the destination partition. After the Leader writes the message to its local disk log, the Leader sends a success response to the producer.

  • Analogy: Sending mail with a standard courier receipt.
  • Advantage: Guarantees data is safely stored on at least one server.
  • Disadvantage: Still a data loss risk. If the Leader broker crashes before the Follower replicas copy the message, and one Follower is elected as the new Leader, that message is lost.

3. acks = all or acks = -1 (Maximum Safety, Zero Data Loss) #

The producer waits for success confirmation from the Leader broker and all active partition replicas in the In-Sync Replicas (ISR) group. This configuration must be paired with the topic-level min.insync.replicas parameter (at least 2).

  • Analogy: Sending valuable mail with wet signatures from all official witnesses.
  • Advantage: Absolute data safety guarantee. As long as at least one ISR replica stays alive, data is guaranteed never to be lost. Highly recommended for financial transactions, payment systems, and sensitive data audits.

Error Handling and Retry #

Computer networks are unreliable systems. Transient errors like lost socket connections, network route congestion, or leader election processes frequently occur in distributed clusters.

Kafka producers have built-in error handling mechanisms to deal with these temporary failures automatically without burdening our application code:

  • retries: Determines how many times the producer attempts to resend a failed message batch caused by transient errors. Since Kafka 2.0, the default is 2147483647 (unlimited), meaning the producer keeps trying until the wait time runs out.
  • delivery.timeout.ms: The total message delivery time limit (default 120,000 ms or 2 minutes). If a message isn’t successfully sent after this limit, the producer gives up and throws an exception to our application.

The Danger of Out-of-Order from Retry #

One dangerous side effect of message resend attempts is the potential for broken data ordering within a partition.

For example, the producer sends Batch A, then Batch B. Batch A fails due to a temporary network issue, but Batch B is successfully sent. When the producer retries to resend Batch A, Batch A gets written to the partition after Batch B, meaning the order is reversed to B then A.

To absolutely solve this problem, we must set the following configuration:

  • Limit the number of active connections per partition: max.in.flight.requests.per.connection = 1. This forces the producer not to send Batch B before getting certainty about Batch A’s status.

Introduction to the Idempotent Producer #

The most confusing network failure in distributed systems happens when the producer successfully sends a message to the broker, the broker successfully writes the data to disk, but the network connection drops right before the broker sends the success ACK signal back to the producer.

Because it doesn’t receive the ACK, the producer assumes delivery failed and tries to resend the same message. As a result, the same data is written twice in the Kafka partition (data duplication).

Since Kafka 0.11, we can elegantly solve this problem by enabling the Idempotent Producer feature through the configuration parameter:

$$\text{enable.idempotence} = \text{true}$$

Since Kafka 3.0, this idempotence parameter defaults to true.

How Idempotence Works #

When this feature is active, the Kafka broker assigns a unique ID to each producer (Producer ID or PID) and a Sequence Number for each message batch sent.

When the broker receives a message batch, it checks whether that PID and Sequence Number have already been recorded in the partition log. If the sequence number already exists, the broker discards the duplicate batch to keep the data clean, but still sends a success ACK signal back to the producer so it doesn’t resend.

Common Mistakes (Anti-patterns) in Producer Usage #

Here are some common Kafka producer implementation mistakes found in the industry, along with their fixes:

1. Creating a New Producer Instance for Every Message Send #

Developers accustomed to stateless REST API architectures often create a new producer object for every data send request, then close the connection (close()) right after the data is sent.

Consequences: The Producer object in Kafka is a very heavy component. Every time it’s created, it must perform an initial connection (bootstrap), download the entire cluster’s metadata from brokers, allocate a giant memory buffer for the Record Accumulator, and start a new background thread. Creating producers repeatedly exhausts the application server’s memory within seconds, floods the cluster with new TCP connections, and triggers out-of-memory exceptions.

# ANTI-PATTERN: Creating a new producer instance for every event
# This wastes memory and floods the broker with rogue TCP connections.
def kirim_data_boros(data_event):
    from kafka import KafkaProducer
    # CREATING A NEW INSTANCE REPEATEDLY (VERY DANGEROUS!)
    producer = KafkaProducer(bootstrap_servers='localhost:9092')
    producer.send('topik-log', value=data_event)
    producer.close()

# The CORRECT solution: Using the Singleton pattern (One Instance for the Whole Application)
# The producer instance is created once at application startup and shared thread-safely.
class KafkaProducerSingleton:
    _instance = None

    @classmethod
    def get_producer(cls):
        if cls._instance is None:
            from kafka import KafkaProducer
            # Creating a single instance that's used continuously
            cls._instance = KafkaProducer(
                bootstrap_servers='localhost:9092',
                enable_idempotence=True,  # Guarantee idempotent delivery
                linger_ms=20              # Batching optimization for high throughput
            )
        return cls._instance

def kirim_data_aman(data_event):
    # Getting the same single instance
    producer = KafkaProducerSingleton.get_producer()
    producer.send('topik-log', value=data_event)

2. Making Blocking Synchronous Calls (.get()) for Every Message #

Some developers use producer.send(...).get() to ensure a message truly arrives before continuing to the next code execution.

Consequences: Calling .get() forces the asynchronous I/O thread to become synchronous and blocks the main thread execution. This kills all the batching optimization benefits and the Record Accumulator buffer memory in the producer. Our producer throughput drops drastically from tens of thousands of messages per second to just a few hundred per second because it must wait for round-trip network latency for every individual message. Use asynchronous Callbacks to monitor delivery status non-blocking.

Summary #

  • Asynchronous Architecture — The Kafka Producer SDK processes data delivery asynchronously through the Serializer (convert to binary), Partitioner (determine partition route), Record Accumulator (store in buffer memory), and Sender Thread (network sender) components.
  • Batching Optimization — Set the batch.size (maximum batch size) and linger.ms (extra wait time) parameters optimally to balance data throughput and system latency.
  • Acks Configuration — The acks parameter determines the data safety tolerance level: acks=0 (maximum speed, high data loss risk), acks=1 (Leader confirmation only), and acks=all (waits for confirmation from the Leader and the entire ISR group for full safety).
  • Idempotent Delivery — Always enable the idempotent producer feature (enable.idempotence=true) to prevent data duplication from temporary network connectivity issues.
  • Singleton Design Pattern — Use one single producer object instance (singleton) shared cross-thread across the entire application to avoid memory resource and TCP connection leaks.
  • Use Non-blocking Callbacks — Avoid calling the .get() method directly on send operations because it blocks the thread workflow and drastically reduces data throughput performance.

← Previous: Partition Next: Consumer →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact