Acknowledgement #

In distributed system architecture, reliable data delivery requires clear receipt confirmation between the sender (producer) and receiver (broker). In Apache Kafka, this guarantee is controlled by a configuration called Acknowledgement (ACKs). This parameter sets how certain a producer must feel that the message it sent is truly safely stored in the broker cluster before it can continue sending the next message. Choosing the right ACKs value is a critical architectural decision because it determines the most fundamental trade-off in distributed systems: speed (latency/throughput) against data resilience (durability).

Dissecting the Three Levels of acks Configuration #

Kafka producers provide the acks configuration at the client level supporting three value options: 0, 1, and all (or -1). Each option sets a very different success criterion on the broker side.

1. acks=0 (Fire-and-Forget) #

When we set acks=0, the producer considers the message successfully sent immediately after the data is written to the transport network socket, even before the broker receives it. The producer waits for no response or confirmation from the broker.

• How It Works: The producer sends the record to the broker and immediately continues to the next record, not caring whether the broker is down, the disk is full, or there’s a network disruption.
• Advantages: Lowest latency and highest throughput because there’s no network wait time for reply packets (RTT — Round Trip Time is eliminated from the write cycle).
• Weaknesses & Risks: Very high data loss risk. If a failure happens on the receiving broker, the producer never knows, and lost data is never resent (retry won’t trigger because the producer assumes delivery always succeeds).
• Use Case: High-frequency IoT sensor data collection, real-time fleet GPS coordinate tracking, or telemetry logs where losing a few messages doesn’t damage overall business integrity.

2. acks=1 (Leader Acknowledgement) #

The acks=1 value is the middle ground balancing speed and safety. In this mode, the producer waits for success confirmation only from the broker acting as the Leader of the destination partition.

• How It Works: The leader broker receives the message from the producer, writes it to its own local commit log file, then immediately sends a successful ACK to the producer. Replication to follower brokers happens afterward asynchronously.
• Advantages: Relatively low latency because the producer only waits for one local broker to write data to disk/page cache before continuing.
• Weaknesses & Risks: There’s a data loss risk window. If the leader broker experiences a sudden physical failure (crash/hardware failure) right after sending the ACK to the producer, but before the follower brokers manage to fetch that new message, the data is lost. When one follower is elected as the new leader, that data will never be found.
• Use Case: User web activity logs, business application metrics, or analytics tracking where high reliability is needed but a little data loss from total server failure is still tolerable.

3. acks=all or acks=-1 (Full Quorum Acknowledgement) #

The acks=all (or acks=-1) mode offers the highest data durability level. The producer waits until the message is written not only by the leader, but also by all active replicas in the In-Sync Replicas (ISR) group.

• How It Works: The leader broker receives the message from the producer, writes it to the local log, then waits for ISR-registered followers to fetch and write that data to their own logs. After the ISR quorum is met, the leader sends a successful ACK to the producer.
• Advantages: Absolute durability guarantee. As long as at least one active backup replica exists in the ISR, data is guaranteed not to be lost even if the leader broker is physically destroyed.
• Weaknesses & Risks: Highest write latency because the producer must wait for several network communication round trips and I/O operations from multiple brokers at once before getting confirmation.
• Use Case: Financial transactions, e-commerce order processing systems, bank balance change records, compliance audit logs, or any scenario applying a zero data-loss policy.

Visual Flow of the Acknowledgement Process #

To understand the latency differences and safe points of each configuration, let’s compare the message lifecycle from producer to broker through the following sequential diagrams:

flowchart TD
    subgraph Skenario_Acks0["acks=0 Scenario (Fire-and-Forget)"]
        direction TB
        P0["Producer"] -->|"Send Message (Without Waiting)"| L0["Leader Broker"]
        P0 -. "Immediately send the next message" .-> P0
    end

    subgraph Skenario_Acks1["acks=1 Scenario (Leader Only)"]
        direction TB
        P1["Producer"] -->|"Send Message"| L1["Leader Broker"]
        L1 -->|"Write to Local Log"| L1
        L1 -->|"Send ACK (Success)"| P1
        L1 -.->|"Replication (Async)"| F1["Follower Broker"]
    end

    subgraph Skenario_AcksAll["acks=all Scenario (ISR Quorum)"]
        direction TB
        P2["Producer"] -->|"Send Message"| L2["Leader Broker"]
        L2 -->|"Write to Local Log"| L2
        F2["Follower Broker"] -->|"Fetch Data"| L2
        F2 -->|"Write to Follower Log"| F2
        F2 -->|"Send LEO Update"| L2
        L2 -->|"Send ACK after ISR Quorum"| P2
    end

Internal Broker Anatomy: How Is acks=all Processed? #

When a producer sends a write request (produce request) with the acks=all configuration, the broker doesn’t process it simply in one thread. Behind the scenes, Kafka uses an event-driven asynchronous data structure leveraging a component called DelayedProduce.

How Delayed Operation Purgatory Works #

To understand how Kafka manages confirmation delays without blocking main I/O threads, we must look at the broker’s purgatory mechanism:

1. Request Reception: The producer client sends a message to the leader broker. The broker’s I/O thread (KafkaRequestHandler) reads the message from the socket and writes that data to the leader partition’s local log file.
2. ISR Check: The leader broker checks whether the partition’s current replication status meets the criteria. If all ISR-registered followers already have the same (or higher) offset of this new message, the leader immediately replies with success. However, this rarely happens instantly.
3. Storage in Purgatory: If followers haven’t synced yet, the leader wraps the request into a DelayedProduce object and stores it in the Operation Purgatory (a special waiting area for delayed operations).
4. Replica Fetcher: The follower broker keeps sending periodic fetch requests to the leader. When the follower successfully copies the new message, it updates its Log End Offset (LEO) value.
5. Re-evaluation: Once the leader detects that the LEO of all ISR followers has passed that message’s offset, the DelayedProduce operation in purgatory is declared completed.
6. ACK Delivery: The processing thread takes back the completed operation from purgatory and sends a successful ACK response packet to the producer.

This purgatory mechanism is crucial because it ensures the broker’s main I/O handler threads are never blocked waiting for follower network activity. The broker stays free to serve other incoming requests from other clients.

Critical Alliance: acks=all and min.insync.replicas #

Setting acks=all on the producer side is only half the step to securing our data. Full durability guarantees only form when collaborated with a broker/topic-side configuration called min.insync.replicas.

The Single ISR Configuration Trap #

Imagine we have a topic with replication.factor=3 and the producer sends data with acks=all. By default, Kafka’s min.insync.replicas configuration value is 1.

Let’s see what happens when disaster strikes:

1. Two follower brokers experience network failures and leave ISR membership. The remaining ISR member is now only 1 broker (the leader itself).
2. The producer sends a message with acks=all.
3. The leader receives the message, writes it locally, and sees that all current ISR members (which happen to be only itself) have written the data.
4. Because the “all ISR members” criterion is met, the leader immediately sends a successful ACK to the producer.
5. Moments later, that leader broker suddenly dies before the two followers recover.

In this scenario, even though we use acks=all, the data is still lost! This happens because the ISR criterion shrinks to only one broker, so acks=all behavior is automatically downgraded to be equivalent to acks=1.

Industry-Standard Configuration for Maximum Safety #

To prevent the scenario above, we must set the minimum number of active replicas that must exist in the ISR for write operations to be allowed. The safe configuration is as follows:

# Topic (or Cluster) Properties
min.insync.replicas=2

# Producer Properties
acks=all

With this configuration, if the number of active replicas in the ISR drops below 2 (for example, two of three brokers die), the leader broker immediately rejects new messages sent by the producer and replies with the NotEnoughReplicasException or NotEnoughReplicasAfterAppendException error.

Our system detects this rejection and suspends writing to prevent writing data that can’t be safely replicated. This is a real example where we prefer write unavailability to maintain data consistency and durability.

Failure Case Study: Payment Gateway Outage with acks=1 #

Let’s dissect a real-world incident that often happens due to misconfigured ACKs parameters in large-scale e-commerce architecture.

System Scenario #

• Topic: payment-transactions
• Replication Factor: 3
• Initial Configuration: acks=1 (Leader Acknowledgement)
• Incident:
  1. A client sends a payment transaction worth Rp10,000,000.
  2. Broker 1 (Leader) receives the transaction, writes it to the local log, and sends a successful ACK to the producer.
  3. The payment gateway application receives the ACK, records the transaction status as “PAID”, and shows a success page to the customer.
  4. Exactly 20 milliseconds later, before Broker 2 and Broker 3 (Follower) manage to fetch that transaction data, Broker 1 experiences a hardware failure (kernel panic from RAM failure).
  5. The cluster supervisor (KRaft Controller) detects Broker 1’s death and elects Broker 2 as the new Leader for that partition.
  6. Because Broker 2 didn’t get the chance to receive that transaction’s replica before Broker 1 died, the Rp10,000,000 transaction is permanently lost from the Kafka cluster.

Business Impact #

• Data Mismatch: Customers have lost their money because the external system declared the payment successful, but the order processor reading from Kafka never saw the payment event, so the order wasn’t fulfilled.
• Financial & Reputation Loss: The operations team had to do days-long manual reconciliation matching external database logs with Kafka logs to find the lost transaction.

Engineering Solution #

The architecture team changed the configuration to:

• Producer: acks=all
• Topic: min.insync.replicas=2

Now, if the same incident repeats, transactions won’t be reported successful to customers unless at least 2 brokers hold copies of the data. If one dies, the remaining broker is guaranteed to have identical data.

Qualitative Comparison Analysis #

To help us choose the configuration most relevant to our business needs, the table below summarizes the performance characteristics of each acks level:

Failure Handling & Error Codes on the Client Side #

When a producer uses the acks=1 or acks=all parameters, it must be ready to handle various error responses thrown by the broker if a write failure occurs. Here are the main exceptions our application code must manage:

1. NotEnoughReplicasException #

This exception is thrown by the leader broker when it receives a write request with acks=all, but the current number of active replicas in the ISR is less than the min.insync.replicas parameter value.

• Cause: A follower broker crashed, lagged severely, or had inter-broker network connection issues, leaving ISR members below the safe limit.
• Application Solution: The producer must catch this error, temporarily stop sending data to that partition, and trigger an alerting mechanism because cluster health is in a critical condition.

2. NotEnoughReplicasAfterAppendException #

Almost similar to the previous error, but happens after the leader successfully writes the message to its local log before realizing replicas can’t be synced to the minimum ISR follower count.

• Cause: A follower’s sudden death right as the leader’s local write is happening.
• Application Solution: This error requires an idempotent retry to ensure data isn’t duplicated when the cluster recovers.

Example Error Handling Implementation in Java #

Here’s an example of writing robust producer code in Java to handle the architectural errors above using asynchronous callbacks:

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import org.apache.kafka.common.errors.NotEnoughReplicasException;
import org.apache.kafka.common.errors.TimeoutException;

public class HighDurabilityProducer {
    private final KafkaProducer<String, String> producer;

    public HighDurabilityProducer(Properties configs) {
        this.producer = new KafkaProducer<>(configs);
    }

    public void sendTransaction(String key, String transactionData) {
        ProducerRecord<String, String> record = new ProducerRecord<>("payment-transactions", key, transactionData);
        
        producer.send(record, (RecordMetadata metadata, Exception exception) -> {
            if (exception != null) {
                // Checking the error type specifically
                if (exception instanceof NotEnoughReplicasException) {
                    System.err.printf("[DANGER] Failed to write transaction %s. ISR replica count is below the minimum limit!%n", key);
                    logToAlternativeStorage(key, transactionData); // Secure data to local storage media (fallback)
                } else if (exception instanceof TimeoutException) {
                    System.err.printf("[WARNING] Timeout while waiting for ACK for transaction %s. Triggering retry.%n", key);
                } else {
                    System.err.printf("[ERROR] Failed to send message. Reason: %s%n", exception.getMessage());
                }
            } else {
                System.out.printf("[SUCCESS] Transaction %s successfully recorded on partition %d, offset %d%n", 
                    key, metadata.partition(), metadata.offset());
            }
        });
    }

    private void logToAlternativeStorage(String key, String data) {
        // Implement temporary local storage (e.g., local SQLite or disk log)
        // so data isn't lost when the Kafka cluster rejects writes.
    }
}

Complete Client Configuration Property Examples #

Here are two producer configuration profiles in Java Properties optimized for two different extreme scenarios.

1. Financial Transaction Profile (Absolute Durability, Zero Data Loss Tolerance) #

Use this profile for payment modules, ledger transaction records, order processing, or comparative database synchronization.

# Demands acknowledgement from the entire active replica quorum
acks=all

# Enabling the idempotent producer to prevent data duplication from retries
enable.idempotence=true

# Unlimited resend attempt limit (retry until success)
retries=2147483647

# Keeping parallel delivery ordering safe
max.in.flight.requests.per.connection=5

# Acknowledgement wait timeout from the broker (30 seconds)
request.timeout.ms=30000

2. IoT Telemetry Profile (Maximum Throughput, High Data Loss Tolerance) #

Use this profile for server infrastructure metric ingestion, non-critical debug logs, user mouse movement tracking (clickstream), or periodic weather sensor readings.

# Fire-and-forget mode for maximum speed
acks=0

# Disabling idempotence because no ACK is waited for (reduces memory overhead)
enable.idempotence=false

# Reducing retries because if it fails, the next message is more valuable than the old one
retries=0

# Optimizing compression for large throughput
compression.type=lz4

# Collecting as many messages as possible before sending
linger.ms=20
batch.size=65536

Summary #

• acks Parameter: Controls the message delivery success criterion from the producer to the Kafka cluster based on broker confirmation.
• Value Options:
  • acks=0: Maximum speed, no verification, highest data loss risk.
  • acks=1: Medium speed, one broker (leader) verification, data loss risk if the leader crashes before replication.
  • acks=all: Maximum safety, full ISR quorum verification, highest latency.
• Purgatory Mechanism: The broker uses delayed operations (DelayedProduce) to wait for follower replication asynchronously without blocking main processing threads.
• Important Synergy: The acks=all configuration can’t protect data if min.insync.replicas isn’t set with discipline (recommended at least 2 for replication factor 3).
• Business Trade-off: Choose the acks level based on our business data sensitivity; don’t force acks=all for ordinary application logs, and don’t use acks=1 for financial transactions.