Acks, Retries, & Linger.ms #

When we build producer client applications to send messages to Apache Kafka, the default configuration provided by the client library doesn’t always match our system’s needs. Kafka is designed to serve wildly different system requirements: from IoT sensor recording systems needing millions of messages per second without caring about losing a small fraction of data, to banking financial transaction systems demanding absolute reliability without losing a single message, even if it means sacrificing a little speed. The two main pillars determining this behavior are in our hands through producer parameter tuning. Understanding how to tune data resilience parameters like acks and retries, as well as throughput optimization parameters like linger.ms and batch.size, is an essential skill for designing resilient and efficient data architectures.

Dissecting the Data Resilience Configuration: acks #

The acks (acknowledgement) parameter controls how many brokers in the cluster must receive a message and send confirmation back to the producer before that message delivery is considered successful. Tuning this parameter is a direct trade-off between data durability and delivery latency.

There are three configuration values we can give the acks parameter:

1. acks=0 (No Confirmation) #

The producer is considered to have successfully sent the message as soon as it writes it to the TCP network socket. The producer doesn’t wait for any confirmation from the Kafka broker.

• Advantages: Very low transmission latency and maximum throughput because the producer isn’t blocked by broker I/O processes.
• Disadvantages: Very high data loss risk. If the destination broker dies, or the disk fills up right after the message enters the socket, the producer never knows the message was lost and keeps sending the next one.
• Use Case: Non-critical server log collection, real-time metric tracking.

2. acks=1 (Leader Confirmation Only) #

The producer waits for a successful write confirmation only from the broker acting as the Leader of the destination partition. The Leader writes the data to its local commit log (guaranteeing disk storage) before sending a successful ACK back to the producer.

• Advantages: Provides a moderate balance between fast throughput and reasonable data durability guarantees.
• Disadvantages: Still has a data loss window. If the leader dies right after sending a successful ACK to the producer, but before Follower brokers get a chance to replicate the data, that data is permanently lost when one of the lagging followers is promoted as the new leader.
• Use Case: User click behavior tracking systems (clickstream), non-financial statistics reporting.

3. acks=all or acks=-1 (Full ISR Confirmation) #

The producer only considers message delivery successful after the partition leader receives confirmation from all replicas in the In-Sync Replicas (ISR).

• Advantages: Absolute data reliability. As long as at least one backup ISR broker stays alive, the message is guaranteed not to be lost from the cluster.
• Disadvantages: Highest delivery latency because the producer is bound by inter-broker network replication sync time.
• Use Case: Financial payment transactions, invoice generation, security audit records.

[!WARNING] The acks=all property doesn’t provide full data safety guarantees if the min.insync.replicas parameter on the broker topic side is set to 1. If min.insync.replicas=1, then even though the producer requests acks=all, the leader broker immediately sends a successful ACK even if there are no other active followers replicating the data (because the leader considers itself sufficient to meet the minimum 1 ISR requirement). For financial reliability, always set min.insync.replicas=2 on brokers with a Replication Factor of 3.

Handling Network Disruptions: retries and Timeout Parameters #

Network connections between producer applications and Kafka brokers aren’t always stable. Temporary network glitches, new leader election processes, or rolling restart broker processes can trigger message delivery failures. This is where error resilience parameters kick in.

1. The retries Property #

This parameter determines how many times the producer retries sending a message that failed due to temporary problems (retriable errors like network timeouts or the broker leader is switching).

• Modern Configuration: In the latest Kafka producer client versions, the retries default is set to the maximum integer value (Integer.MAX_VALUE). The producer keeps trying to resend the message until the overall duration limit expires.

2. The retry.backoff.ms Property #

By default, if a delivery fails, the producer immediately retries. However, bombarding a busy or dead broker with continuous requests without pauses can worsen that broker’s condition.

• How It Works: The retry.backoff.ms parameter (default: 100 ms) gives the producer a rest pause (wait time) before the next retry attempt, giving the broker cluster time to recover.

3. The delivery.timeout.ms Property #

This is the most important total time limit on the producer side. The delivery.timeout.ms property (default: 120,000 ms or 2 minutes) limits the total duration from the .send() call until the producer receives a successful ACK or gives up and throws an error exception.

• Timeout Formula: The delivery.timeout.ms value must always be greater than or equal to the sum of the request.timeout.ms and linger.ms parameters:

$$\text{delivery.timeout.ms} \ge \text{request.timeout.ms} + \text{linger.ms}$$

If a message doesn’t receive a successful ACK within the default 2-minute window (because the network is completely down), the message is discarded from the Record Accumulator queue and our application receives a delivery failure.

Throughput vs Latency Tuning: linger.ms and batch.size #

The asynchronous message delivery cycle in Kafka producers heavily relies on efficient message grouping in the Record Accumulator memory buffer. We can boost our application’s throughput performance by tuning these two configurations:

1. The batch.size Property #

Determines the maximum memory capacity (in bytes) allocated to hold per-partition messages in the Record Accumulator. The default is 16,384 bytes (16 KB).

• Optimization: If our messages are large or our application throughput is very dense, raising batch.size to 64 KB or 128 KB is highly recommended so the buffer doesn’t fill up too fast and trigger repeated small network packet sends.

2. The linger.ms Property #

Determines how long (in milliseconds) the producer delays sending batches in memory before being taken by the Sender Thread. By default, this property is set to 0 ms (the producer sends messages as soon as possible without waiting for other messages to join the same batch).

• Optimization: By adding a small wait time, for example linger.ms=20 (20 milliseconds), we give the main application thread a chance to stack more messages into the same batch before the Sender Thread sends it to the socket. This multiplies throughput with just 20 milliseconds of added latency overhead.

Mermaid Diagram: The Relationship of Linger.ms to Batch Transmission #

Here’s a visualization of how the linger.ms and batch.size parameters collaborate in the Record Accumulator to control the Sender Thread’s decision before sending data to the broker:

flowchart TD
    subgraph Accumulator["Record Accumulator (Memory Buffer)"]
        direction TB
        Rec1["Message 1 (T=0ms)"] --> B1["Partition 0 Batch"]
        Rec2["Message 2 (T=2ms)"] --> B1
        Rec3["Message 3 (T=5ms)"] --> B1
    end
    
    subgraph Decision["Sender Thread Decision Loop"]
        direction TB
        Cond1{"Batch Size >= batch.size?"}
        Cond2{"Held Time >= linger.ms?"}
    end
    
    B1 --> Cond1
    Cond1 -- "Yes (Batch Full, e.g., 16KB)" --> SendNow["Send Immediately to Network"]
    Cond1 -- "No" --> Cond2
    
    Cond2 -- "Yes (e.g., linger.ms=20ms elapsed)" --> SendNow
    Cond2 -- "No" --> Wait["Hold in Buffer (Accumulate New Messages)"]
    
    SendNow --> Broker["Kafka Broker"]
    
    style Accumulator stroke:#e5e7eb
    style Decision stroke:#e5e7eb
    style B1 stroke:#0288d1,stroke-width:2px
    style SendNow stroke:#2e7d32,stroke-width:2px
    style Wait stroke:#f57c00,stroke-width:2px

Max In-Flight Requests Interaction with Message Ordering #

One of the most important parameters often overlooked when designing producer stability is max.in.flight.requests.per.connection (default: 5). This parameter determines how many unacknowledged produce requests (not yet getting a successful ACK from the broker) a producer may send through one TCP socket connection simultaneously.

Why Does This Property Affect Data Ordering? #

If this property is set to a value greater than 1 (for example, using the default 5) and we leave enable.idempotence=false, we risk out-of-order messages if a temporary network failure occurs.

Let’s simulate the following delivery failure scenario:

1. The producer sends Batch A to the broker. Since there’s no ACK yet, Batch A is in-flight.
2. Without waiting for Batch A’s ACK, the producer sends Batch B asynchronously. Now there are 2 in-flight requests.
3. Batch A fails to be received by the broker due to a temporary network disruption (for example, the leader broker is busy electing a new leader).
4. Batch B is successfully received by the broker and written to the partition commit log.
5. The producer receives a failure signal for Batch A. Because the retries parameter is active, the producer retries sending Batch A.
6. The Batch A resend attempt succeeds and is written to the broker.

• Destructive End Result: In the broker commit log, Batch B is written before Batch A. The message order is permanently reversed!

Solutions for Message Ordering Problems #

• Old Way (Before Idempotency): We’re forced to set max.in.flight.requests.per.connection=1. This configuration forces the producer to act synchronously — waiting for Batch A’s ACK before Batch B can be sent. However, this limits delivery throughput because network bandwidth is never fully utilized.
• Modern Way (Recommended): We keep the max.in.flight.requests.per.connection value between 1 and 5, and must set enable.idempotence=true. Idempotence uses sequence numbers to track batch order on the broker side. The broker refuses to write Batch B if it detects the preceding sequential Batch A hasn’t arrived yet, forcing the broker to correctly reassemble scrambled batches before writing to disk.

Data Compression on Producers: Gzip, Snappy, Lz4, vs Zstd #

By default, messages are sent from producers to brokers as uncompressed raw data. If our application processes millions of medium-to-large messages daily, enabling data compression via the compression.type property on the producer side is a very significant optimization step.

Kafka End-to-End Compression Advantages #

Kafka’s compression cycle is very efficient because it operates end-to-end:

1. Producers compress the data batch.
2. Data is sent as compressed binary over the network, saving bandwidth.
3. Brokers store the compressed bytes directly to disk without decompressing (saving broker memory and CPU).
4. Consumers receive the compressed bytes and decompress them on their application side.

Compression Algorithm Comparison #

For most common production applications, choosing snappy or lz4 is the safest decision to minimize CPU load on our producer applications, while zstd is the ideal choice if our network bandwidth is very limited but we still have adequate spare CPU compute power.

Java Configuration Implementation: High-Throughput vs High-Durability #

As system architects, we must be able to configure producers to match our business data characteristics. Here’s a comparison code between a wrong producer configuration (anti-pattern) and optimal configurations for two different use cases:

// ANTI-PATTERN: Default or directionless mixed configuration
// Unsafe financial transaction setup because acks=1 and retries are disabled
public class NaiveFinancialProducer {
    public Properties getProperties() {
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9092");
        
        // ✗ Very dangerous for financial transactions: data can be lost if the leader restarts
        props.put("acks", "1"); 
        
        // ✗ Disabling auto retry forces the app to manually handle transient errors
        props.put("retries", "0"); 
        
        // ✗ linger.ms too high without clear reason in a latency-sensitive transactional system
        props.put("linger.ms", "5000"); 
        return props;
    }
}

// CORRECT: Scenario 1 Configuration - High Durability (Financial Grade, No Data Loss)
public class HighDurabilityProducer {
    public Properties getProperties() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        
        // ✓ Absolute reliability: Waiting for confirmation from the entire ISR
        props.put(ProducerConfig.ACKS_CONFIG, "all"); 
        
        // ✓ Auto retry until the time limit expires to avoid transient data loss
        props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
        
        // ✓ Enable producer idempotence to prevent duplication from network retries
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); 
        
        // ✓ Limit in-flight requests to 5 to guarantee data ordering and idempotence compatibility
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
        
        // ✓ Overall record submission time limit before an error is thrown to the application
        props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, "120000"); // 2 minutes
        return props;
    }
}

// CORRECT: Scenario 2 Configuration - High Throughput (Massive Logging/IoT Clickstream)
public class HighThroughputProducer {
    public Properties getProperties() {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        
        // ✓ Only wait for leader confirmation to save RTT transmission latency
        props.put(ProducerConfig.ACKS_CONFIG, "1"); 
        
        // ✓ Add a 20ms pause so the application thread can stack messages into one batch
        props.put(ProducerConfig.LINGER_MS_CONFIG, "20"); 
        
        // ✓ Raise the memory batch capacity to 64KB (from the default 16KB)
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, String.valueOf(64 * 1024)); 
        
        // ✓ Enable binary data compression (e.g., Snappy or Zstd) to save bandwidth
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy"); 
        
        // ✓ Allocate a wider buffer pool memory to avoid backpressure
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, String.valueOf(64 * 1024 * 1024)); // 64 MB
        return props;
    }
}

Tuning Recommendations for Various Use Cases #

To make decision-making easier when designing production systems, here’s a configuration parameter reference matrix based on use case:

Summary #

• ACKS Levels: Tuning the acks property controls the broker confirmation level (0, 1, all) determining the latency vs data reliability trade-off.
• min.insync.replicas: The acks=all value on producers requires the min.insync.replicas >= 2 configuration on the broker side to guarantee data replication to followers.
• Linger & Batch: Combining the linger.ms (e.g., 20ms) parameter with batch.size (e.g., 64KB) can cut network latency and multiply throughput.
• Idempotence: Always enable enable.idempotence=true when using high retries to safely discard duplicate data copies on the broker side.
• Delivery Timeout: The delivery.timeout.ms parameter limits how long the producer tolerates retry attempts before throwing an exception to the main application thread.

← Previous: Key vs No-Key Message Next: Idempotent Producer →