Idempotent Producer #

In distributed system architecture, network failure isn’t a possibility anymore — it’s a certainty that will happen sooner or later. When a producer application sends a message to Apache Kafka, network disruptions can happen mid-transmission. The scenario that most often becomes data architects’ nightmare is when the broker successfully receives and writes data to disk, but the TCP connection suddenly drops right before the broker can send the confirmation signal (acknowledgement/ACK) back to the producer. Because the producer doesn’t receive the success confirmation, it performs a retry. Without a special handling system, the broker writes the same message a second time. This data duplication problem can totally destroy our business system integrity, especially in sensitive transactions like e-commerce payments or bank balance tracking. This is where the importance of enabling the Idempotent Producer feature (enable.idempotence=true) comes in, providing precise data delivery guarantees without duplication.

The Data Duplication Problem in Distributed Systems #

Before dissecting the solution Kafka offers, we need to understand the anatomy of message duplication from retries.

In distributed system data transmission theory, there are three types of delivery semantics:

  1. At-Most-Once (Maximum Once): Messages are sent once with never any retry attempts if it fails. There will never be duplicate data, but there’s a data loss risk.
  2. At-Least-Once (Minimum Once): The producer keeps sending the same message until it obtains a successful ACK from the broker. Data is guaranteed never lost, but there’s a data duplication risk if the ACK is lost in transit.
  3. Exactly-Once (Exactly Once): Messages are guaranteed to reach the broker exactly once, no less and no more, even with repeated network failures.

By default, without idempotence, Kafka producers operate with the At-Least-Once model.

The Anatomy of Message Duplication #

  1. Our application thread calls producer.send(Pesan_A).
  2. The producer client sends the data through a TCP network socket to the partition leader broker.
  3. The broker successfully receives Pesan_A, writes it to the physical commit log file on local disk, and updates the partition offset (for example, to Offset 100).
  4. The broker prepares a successful ACK response packet and sends it back to the producer.
  5. Disruption Happens: Right before the ACK packet reaches the producer, the internet connection experiences a temporary disruption (packet loss). The connection socket drops.
  6. The producer client detects the lost connection. Because the retries property is active, the producer triggers an automatic resend attempt.
  7. The producer sends Pesan_A data again to the broker.
  8. The broker (without the idempotence feature) receives Pesan_A as a new independent message, writes it to the physical commit log at Offset 101, and sends a successful ACK that’s finally received by the producer.
  9. In the downstream database, our transaction is recorded twice at offsets 100 and 101.

Kafka’s Solution: Idempotent Producer #

To permanently solve this problem without destroying network throughput performance, Apache Kafka introduced the Idempotent Producer feature. Starting from Kafka version 3.0, this feature is enabled by default (enable.idempotence=true).

Idempotence ensures that even if the producer sends the same message multiple times to the broker due to network connection failures, the broker only writes that message to disk exactly once at the destination partition offset.

Main Advantages of Idempotence: #

  • High Data Integrity: Eliminates the need to write complex deduping logic at the downstream consumer application level.
  • Maximum Performance: The duplicate filtering process happens at broker memory level with nearly imperceptible CPU overhead (O(1) lookup).
  • Guaranteed Message Order: Idempotence automatically prevents batch data order swapping in partition queues.

Internal Working Mechanism: Producer ID and Sequence Number #

How does the Kafka broker recognize that an incoming message is a re-copy of a previously written message? Kafka implements a binary identity-based tracking mechanism using two main components: Producer ID (PID) and Sequence Number.

1. Producer ID (PID) Initialization #

When a KafkaProducer object first starts in our application and makes an initial connection to the broker coordinator, the producer client sends a special request called InitProducerId.

  • The broker allocates a unique Producer ID (PID) as a 64-bit long integer for that producer session. This PID is stored in broker memory and bound to our producer client connection.

2. Sequence Number Assignment #

Every time the producer prepares a message batch to send to a specific topic partition, the producer attaches a sequence number called a Sequence Number starting from 0.

  • This sequence number increments sequentially specifically for the PID + Specific Topic Partition combination.
  • If we send 3 message batches to Partition 0, those batches are labeled Sequence 0, Sequence 1, and Sequence 2. If we send a batch to Partition 1, the sequence numbering restarts from 0 specifically for Partition 1.

3. Validation Logic on the Broker Side #

When the broker receives a write request from the producer, it checks the batch’s PID and Sequence Number in its memory:

  • New Write Scenario (New_Seq == Last_Seq + 1): If the incoming sequence number (Seq 1) is exactly one number above the last sequence recorded on the broker for that PID (Seq 0), the broker accepts the data, writes it to disk, and updates its memory state to Last_Seq = 1.
  • Duplicate Scenario (New_Seq <= Last_Seq): If the incoming sequence number (Seq 0) is less than or equal to the last successfully written sequence (Seq 0), the broker immediately concludes this data is a duplicate from a previous network failure. The broker silently discards the message (doesn’t write it to disk), but still sends a successful ACK response back to the producer so the producer can safely end its retry cycle.
  • Missing Message Scenario (New_Seq > Last_Seq + 1): If the incoming sequence number jumps (for example, Seq 3 arrives when Last_Seq is only 1), the broker realizes a message in the middle was lost due to transmission failure. The broker rejects the request and returns the fatal error OutOfOrderSequenceException to maintain data order consistency.

Sequence Diagram: Retry Handling with Idempotency #

The following diagram illustrates the stark contrast between a system without idempotence and a system using idempotence when handling lost network ACK confirmation packets:

sequenceDiagram
    autonumber
    actor App as Application Thread
    participant Prod as Producer Client
    participant Broker as Kafka Broker Leader
    
    Note over App, Broker: "Scenario 1: Without Idempotence (Duplication Happens)"
    App->>Prod: send(Message A)
    Prod->>Broker: ProduceRequest (Message A)
    Broker->>Broker: Write Message A to Disk (Offset 100)
    Note right of Broker: "ACK lost in network"
    Broker--xProd: ACK (Delivery Failed)
    Note left of Prod: "Request timeout expired, triggering Retry"
    Prod->>Broker: ProduceRequest (Message A - Resend)
    Broker->>Broker: Write Message A to Disk (Offset 101 - DUPLICATE!)
    Broker-->>Prod: Successful ACK
    Prod-->>App: Success Callback
    
    Note over App, Broker: "Scenario 2: With Idempotence (enable.idempotence=true)"
    App->>Prod: send(Message B)
    Prod->>Broker: ProduceRequest (PID=1001, Seq=0, Message B)
    Broker->>Broker: Write Message B to Disk (Offset 102)
    Note right of Broker: "State: PID=1001, LastSeq=0"
    Note right of Broker: "ACK lost in network"
    Broker--xProd: ACK (Delivery Failed)
    Note left of Prod: "Triggering Automatic Retry"
    Prod->>Broker: ProduceRequest (PID=1001, Seq=0, Message B - Resend)
    Broker->>Broker: "Check State: Seq 0 <= LastSeq 0 (DUPLICATE!)"
    Note right of Broker: "Message silently discarded from disk write"
    Broker-->>Prod: Successful ACK (Indicating data safely stored)
    Prod-->>App: Success Callback

Idempotence Configuration and Compliance Requirements #

To safely enable the idempotent producer feature, several supporting configuration parameters must be aligned. If these configurations contradict each other, the Kafka client throws an exception during application initialization.

Here are the mandatory properties that must be met:

  1. enable.idempotence=true: Enables the Producer ID and Sequence Number logic.
  2. acks=all (or acks=-1): Idempotence requires maximum data resilience so sequence number state stays consistent across all replicas (ISR) if the leader suddenly dies.
  3. retries > 0: The producer must be allowed to make retry attempts to recover from network failures independently.
  4. max.in.flight.requests.per.connection <= 5: The Kafka broker only has a memory cache window to monitor sequence numbers of at most 5 simultaneously running requests per connection. If set above 5, the broker can’t guarantee duplicate detection consistency.

Java Implementation: Enabling the Idempotent Producer #

Here’s a Java code comparison showing how to configure the wrong producer (without idempotence for sensitive transactions) versus the correct way using the Idempotent Producer:

// ANTI-PATTERN: Running transactional message delivery without idempotence
// Prone to duplicating payment data if the server's internet connection is unstable
public class VulnerablePaymentProducer {
    public KafkaProducer<String, String> createProducer() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        
        // ✗ Disabling idempotence makes retries prone to duplicating data
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "false"); 
        props.put(ProducerConfig.ACKS_CONFIG, "1"); // Only waiting for the leader
        props.put(ProducerConfig.RETRIES_CONFIG, 3);
        
        return new KafkaProducer<>(props);
    }
}

// CORRECT: Configuring an idempotent producer for maximum data integrity
public class SecurePaymentProducer {
    public KafkaProducer<String, String> createProducer() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
        
        // ✓ CORRECT: Enable producer idempotence
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); 
        
        // The following properties are auto-aligned by the Kafka Client v3.0+,
        // but writing them explicitly is highly recommended as architectural documentation:
        props.put(ProducerConfig.ACKS_CONFIG, "all"); 
        props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
        
        return new KafkaProducer<>(props);
    }
}

Physical Limitations of the Idempotent Producer #

Although the Idempotent Producer feature is very powerful at solving data duplication, we must understand the technical limitations of this mechanism so we don’t mis-expect its behavior in production.

1. Only Applies to the Same Producer Session (Single Session) #

Kafka idempotence is based on the Producer ID (PID) created when the KafkaProducer object is initialized.

  • If our producer application crashes, exits the JVM, then restarts, the new KafkaProducer object instantiation requests a new PID from the broker (e.g., PID 2005).
  • Because the PID changed from 1001 to 2005, the Kafka broker won’t recognize the old session’s sequence numbers. If the new producer resends a message that the old producer failed to send before the crash, the broker writes it to disk as a new message.
  • Solution: To handle cross-session duplication or crash failures, we must use the advanced level: Transactional Producer (Exactly-Once Semantics) using the persistent transactional.id property across restarts.

2. Limited to the Partition Level (Partition-Level Only) #

The sequence number is bound to the combination of PID and the destination physical partition number.

  • If for some reason (for example, a custom partitioner logic error) the same message is resent to a different partition (e.g., from Partition 0 to Partition 1), the broker can’t detect that duplication because the sequence number on Partition 1 is evaluated independently.

Idempotence State Synchronization During Broker Failover #

One of the main advantages of Kafka’s idempotence architecture is its resilience to broker failures (broker resilience). What happens if the leader broker storing our producer’s Last_Seq memory state suddenly dies or crashes? Is the sequence number data lost, triggering data duplication when a new leader is elected?

The answer is no. Kafka doesn’t just store the PID and Sequence Number state map in the leader broker’s RAM. This state is persisted directly in the header files of every commit log batch (log record batch headers) stored to disk.

Batch Header Storage Structure #

Every time a producer writes a data batch to the broker, that batch is wrapped with header metadata including:

  • Producer ID (PID): The sender’s identity.
  • Producer Epoch: A short integer number preventing stale (zombie producers) from sending old data.
  • Base Sequence Number: The starting sequence number of the messages in that batch.

Replica and Recovery Process (Failover Restoration) #

  1. When the leader broker writes a message batch to its local disk, this header metadata is also permanently written.
  2. Follower brokers replicate this raw binary batch entirely, including its header files, to their respective local disks.
  3. If the leader broker experiences a physical failure (dies), one of the followers in the In-Sync Replicas (ISR) is promoted by the Controller as the new leader.
  4. This new leader scans the active log segment and instantly reads the last batch header to rebuild the Last_Seq memory state map for every active PID.
  5. When our producer retries to the new leader, the new leader already has complete last sequence information and can accurately filter duplicates without any disruption.

Monitoring Idempotence Metrics via JMX #

To ensure the Idempotent Producer feature runs optimally in production, we must monitor the following JMX metrics on our producer clients:

1. Record Retry Rate (record-retry-rate) #

Measures the average number of message delivery retry attempts per second made by the producer.

kafka.producer:type=producer-metrics,client-id=[clientId],name=record-retry-rate
  • Analysis: If this metric value spikes sharply but the data duplication rate in our consumer database stays at zero, that proves network failures are happening but the idempotence feature is working perfectly filtering duplicates on the broker side.

2. Record Error Rate (record-error-rate) #

Measures the average message delivery failures per second caused by fatal errors.

kafka.producer:type=producer-metrics,client-id=[clientId],name=record-error-rate
  • Analysis: A spike in this metric can indicate sequence number mismatch problems (like OutOfOrderSequenceException) requiring deep investigation into network stability or broker replication behavior.

Summary #

  • At-Least-Once Duplication: Network ACK packet delivery failures trigger automatic producer retries, which can duplicate data in the broker log.
  • Idempotent Producer: Enabling the enable.idempotence=true property guarantees that repeatedly sent messages are only written once on the broker.
  • Producer ID & Sequence: The duplicate identification mechanism based on a unique PID per producer session and sequentially incrementing Sequence Numbers per partition.
  • Silent Drop: The broker silently discards duplicate messages with old sequence numbers, but still sends a successful ACK back to the producer.
  • Configuration Requirements: Idempotence requires aligning supporting configurations: acks=all, retries > 0, and max.in.flight.requests.per.connection <= 5.
  • Single Session: Basic idempotence only protects data duplication during the same producer instance session, not covering producer restart/crash scenarios.

← Previous: Acks, Retries, & Linger.ms Next: Exactly-Once Semantics →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact