At-Most-Once #

In distributed system architecture, guaranteeing that data is delivered and processed correctly is one of the biggest challenges. Apache Kafka offers several data delivery guarantee levels known as Delivery Semantics. One of the most fundamental guarantee models is At-Most-Once. In this model, a message is sent or processed at most once. This means a message could be lost mid-way and never arrive or be processed by the downstream system, but that message is guaranteed to never be processed more than once. This guarantee is ideal for scenarios where data duplication is far more dangerous or expensive than losing a few data rows, or when ultra-low latency is our system’s top priority.

Basic Concepts and Philosophy of At-Most-Once #

The At-Most-Once delivery semantic operates with a fire-and-forget philosophy on the producer side, or commit-first on the consumer side. To understand why this guarantee level exists, we must understand the fundamental trade-off in distributed systems between reliability (durability), speed (throughput/latency), and consistency.

In the Kafka ecosystem, by default, if we don’t do special configuration, the system operates close to At-Least-Once. However, for certain applications, the cost of detecting, filtering, and handling message duplication is very high. Duplicate message processing can damage internal application state or cause inconsistencies if the operations performed aren’t idempotent. In such conditions, system developers often choose to sacrifice a small portion of lost messages to maintain processing speed and application logic simplicity without duplication handling overhead.

sequenceDiagram
    autonumber
    participant Prod as Producer (App)
    participant Broker as Kafka Broker
    participant Cons as Consumer (App)

    Note over Prod, Broker: "Producer Side: acks=0 (Fire and Forget)"
    Prod->>Broker: ProduceRequest (Message A)
    Note over Prod: Producer immediately considers it successful without waiting for a response
    Broker--xProd: ACK Lost / Broker Crash (Message A physically lost)

    Note over Broker, Cons: "Consumer Side: Commit-First (At-Most-Once)"
    Cons->>Broker: poll() fetches message B (Offset 200)
    Cons->>Broker: CommitOffsetRequest (Offset 200)
    Broker-->>Cons: Commit ACK (Offset 200 stored)
    Note over Cons: Consumer starts processing message B
    Note over Cons: CRASH! (OOM / Power outage while processing)
    Note over Cons: Consumer restart -> Reading from Offset 201 (Message B lost!)

The diagram above illustrates how data can be lost on both sides (producer and consumer) when we implement At-Most-Once semantics. On the producer side, data loss happens because we don’t wait for confirmation from the broker. On the consumer side, data is lost because we mark that data as “finished processing” (via offset commit) before our business logic actually succeeds in executing it.

Producer-Side Configuration (Producer Client) #

To force the Kafka producer to operate in At-Most-Once mode, we must disable all retries mechanisms and not request receipt confirmations from the broker. The main configuration to set on the producer properties includes:

1. acks=0 #

The acks (acknowledgements) property controls how many broker replicas must receive the message before the producer considers the delivery successful.

  • How It Works: When set to 0, the producer sends a ProduceRequest to the broker and immediately returns a success status to the main application thread without waiting for any confirmation from the leader broker.
  • Impact: If the leader broker crashes right after the data packet arrives at the network socket but before it’s written to the disk log, that message is lost forever. However, delivery latency becomes very low because there’s no network round-trip wait time for the ACK.

2. retries=0 #

By default, the Kafka client library tries to resend messages automatically if a temporary failure happens (like network errors or new partition leader elections).

  • How It Works: By setting retries=0, we forbid the producer from resending on error.
  • Impact: If a momentary network connection disruption happens, the producer immediately gives up and throws an exception to the application, or ignores it depending on our callback handling. This ensures there will be no resends that could potentially produce duplicates on the broker from lost ACKs.

3. max.in.flight.requests.per.connection=1 #

Although retries are already set to 0, limiting the number of requests running on the network in parallel helps maintain message order if unexpected network errors happen on a particular TCP socket connection.

Consumer-Side Configuration (Consumer Client) #

On the consumer side, At-Most-Once semantics is achieved by changing the offset commit operation order. There are two main ways to implement this:

Approach A: Time-based Auto-Commit #

This is the easiest but least precise method. We rely on the consumer’s background thread to send offset commits to the broker periodically without caring about data processing status on the main thread.

Required configuration:

  • enable.auto.commit=true: Enables the automatic commit feature by the consumer.
  • auto.commit.interval.ms=1000: Tells the consumer to send the last fetched offset to the broker every 1 second.

Why does this produce At-Most-Once? When we call the consumer.poll(Duration) function, the consumer fetches a number of messages (e.g., offsets 100 to 150). The background thread automatically sends a commit for offset 150 at the next time interval. If our main application is processing the message at offset 120 and suddenly crashes, offset 150 was already committed to the broker. When the application restarts, it starts reading from offset 151. Messages 121 to 150 are lost without ever finishing processing.

Approach B: Manual Commit Before Processing (Commit-First Pattern) #

This approach gives full control to our code. We disable auto-commit and explicitly do a synchronous commit immediately after messages are successfully fetched from the .poll() function.

Required configuration:

  • enable.auto.commit=false

Operation steps inside the code:

  1. Call consumer.poll() to fetch a data batch.
  2. Call consumer.commitSync() instantly to mark that batch offset as received.
  3. Start running the business logic loop to process the data.
flowchart TD
    Poll["Poll Data"] --> Commit["Commit Offset to Broker"] --> Process["Process Business Logic"] --> Done["Done"]
    Process -- "Application Crash" --> Lost["Data After Commit Is Lost!"]

If the application fails mid-way through step 3, our database or external systems don’t receive that data, but the broker already recorded that the data was consumed. After the system recovers, processing continues to the next batch, avoiding duplication with the risk of losing the failed batch’s data.

Failure Scenario Analysis (Data Loss Deep Dive) #

Let’s dissect chronologically how system failures can cause data loss in an At-Most-Once implementation in production.

Scenario 1: Broker Failure with Producer acks=0 #

  1. The business application calls producer.send(record).
  2. The Kafka client library packages the message and sends it via TCP socket to the Leader Broker IP.
  3. The client immediately returns a successful RecordMetadata object to our application.
  4. At that exact same millisecond, the Leader Broker’s physical server dies suddenly from a power disruption. The message just arrived at the broker server OS kernel memory buffer and hasn’t been copied to the Kafka JVM memory or written to disk using the page cache yet.
  5. The Kafka controller detects the leader’s death and designates a follower broker as the new leader.
  6. Because the follower hasn’t replicated that message yet, the message is lost forever. The producer doesn’t know and won’t resend because of acks=0 and the producer thread has already moved on to the next task.

Scenario 2: Consumer Crash Using the Commit-First Pattern #

  1. The consumer reads 5 records from partition 0: [offset 50, 51, 52, 53, 54].
  2. The consumer immediately calls consumer.commitSync(). The broker stores offset 55 in the internal __consumer_offsets topic.
  3. The consumer starts processing the record at offset 50 (successfully saved to the RDBMS database).
  4. The consumer starts processing the record at offset 51. Mid-processing, the server experiences an Out Of Memory (OOM) error or a network cable disconnects. The JVM thread dies instantly.
  5. After the server restarts, the new consumer instance rejoins the group. The group coordinator reassigns partition 0 to this consumer.
  6. The consumer makes its first .poll() call. The broker sees the last committed offset is 55. The broker sends data starting from offset 55.
  7. Consequence: Records 51, 52, 53, and 54 are never processed again by our application.

Practical Use Cases (Real-World Use Cases) #

The At-Most-Once semantic isn’t a bad pattern; it’s a deliberate architectural choice for specific scenarios. Here are some real use cases where this pattern is highly recommended:

1. User Clickstream Tracking (Clickstream Analytics) #

When we track every user click on an e-commerce site to analyze navigation trends or provide real-time product recommendations. If out of 1,000,000 clicks, 5 clicks are lost from temporary network disruptions, our statistical analysis results won’t change significantly. Conversely, if we use At-Least-Once, duplicate click data scrambles the funnel conversion rate calculation and complicates user page flow visualizations.

2. High-Frequency IoT Sensor Telemetry #

Imagine a temperature sensor on a factory machine sending temperature status every 100 milliseconds. If one temperature data packet is lost at 10:00:00.100, the new data sent at 10:00:00.200 (100 milliseconds later) immediately replaces it. In this scenario, resending stale temperature data (from network delays) would actually scramble real-time analysis and waste bandwidth. We only care about the most current temperature condition.

3. Large-Scale Application Log Aggregation #

Centralized log management systems (like Elasticsearch/Kibana via Logstash or fluentd) process millions of log lines per second from thousands of servers. Losing one debug log line from an application isn’t as critical as a log server experiencing performance bottlenecks from managing failed log resend systems, which can cause server memory blowouts from queue overload (backpressure).

Java SDK Implementation Code: Anti-Pattern vs Safe Solution #

Let’s study Java code implementation examples to understand the safe application of this data delivery semantic and avoid business logic misplacement traps.

1. Producer Side (Java SDK) #

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import java.util.Properties;

public class AtMostOnceProducerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // =====================================================================
        // IMPORTANT AT-MOST-ONCE CONFIGURATION
        // =====================================================================
        // ✓ Set acks to 0 so the producer doesn't wait for broker confirmation
        props.put(ProducerConfig.ACKS_CONFIG, "0");
        
        // ✓ Disable automatic retries to prevent duplicate resends
        props.put(ProducerConfig.RETRIES_CONFIG, 0);
        
        // ✓ Limit in-flight requests to minimize out-of-order delivery risk
        props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);

        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        // ANTI-PATTERN: Using acks=0 for important financial transactions
        // If the network drops, the user's balance decreases in our database but
        // the transfer message never reaches Kafka, and the application doesn't know.
        try {
            String transactionMsg = "{\"userId\":\"usr_99\",\"amount\":150000}";
            // DON'T do this if the message is critical!
            producer.send(new ProducerRecord<>("bank-transfers", "usr_99", transactionMsg));
            System.out.println("✓ Transaction sent (acks=0). Dangerous if the connection is lost!");
        } catch (Exception e) {
            // This catch block is almost never triggered for network errors
            // because send() with acks=0 immediately returns success.
            System.err.println("✗ Error detected at the client local level: " + e.getMessage());
        }

        // CORRECT: Using acks=0 for room temperature sensor data
        // If one data point is lost, there's no bad impact on the AC cooling.
        for (int i = 0; i < 5; i++) {
            String sensorPayload = "{\"temp\": 24.5, \"timestamp\": " + System.currentTimeMillis() + "}";
            producer.send(new ProducerRecord<>("room-temperature", "sensor_01", sensorPayload), new Callback() {
                @Override
                public void onCompletion(RecordMetadata metadata, Exception exception) {
                    if (exception != null) {
                        // This callback is only triggered if there's a local serialization error
                        // before the message touches the network card.
                        System.err.println("✗ Failed to send local sensor metric: " + exception.getMessage());
                    }
                }
            });
        }
        
        producer.close();
    }
}

2. Consumer Side (Java SDK) #

Here’s a consumer implementation using the Commit-First pattern to guarantee At-Most-Once semantics.

import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;

public class AtMostOnceConsumerExample {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "telemetry-aggregator");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        // =====================================================================
        // IMPORTANT AT-MOST-ONCE CONFIGURATION (MANUAL COMMIT FIRST)
        // =====================================================================
        // ✓ Disable auto-commit so we can control when commits are sent
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("room-temperature"));

        try {
            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                
                if (!records.isEmpty()) {
                    // ✓ STEP 1: Immediately commit offsets before processing data!
                    // This establishes the At-Most-Once semantic.
                    try {
                        consumer.commitSync();
                        System.out.println("✓ Offset successfully committed to the broker (Commit-First).");
                    } catch (CommitFailedException e) {
                        System.err.println("✗ Failed to commit offset: " + e.getMessage());
                        // In At-Most-Once, if the commit fails, we may choose to continue
                        // or skip this data to prevent reprocessing.
                        continue; 
                    }

                    // ✓ STEP 2: Process data after the offset is safely committed
                    for (ConsumerRecord<String, String> record : records) {
                        try {
                            // Simulate data processing (e.g., JSON parsing and computation)
                            processTelemetryData(record.value());
                        } catch (Exception e) {
                            // DON'T let an error here trigger re-reading the same message loop.
                            // Because the offset was already committed in step 1, on the next poll iteration
                            // this failed data won't be fetched again by the broker.
                            System.err.println("✗ Failed to process metric at offset " + record.offset() + ": " + e.getMessage());
                        }
                    }
                }
            }
        } finally {
            consumer.close();
        }
    }

    private static void processTelemetryData(String json) {
        // Light telemetry business logic
        System.out.println("Processing telemetry: " + json);
    }
}

When to Choose At-Most-Once? #

To help system architects choose the right semantic, use the following practical guide:

STILL use At-Most-Once if:
  ✓ Ultra-low network transmission latency is an absolute application requirement.
  ✓ Message duplication can cause expensive data damage in downstream databases.
  ✓ Data is continuous with fast value replacement cycles (real-time stream).
  ✓ Our downstream system isn't idempotent and we want to disable retry mechanisms.

DON'T use At-Most-Once if:
  ✗ Every message is high-value and losing even one message can violate financial regulations.
  ✗ We're building an inventory management system, billing system, or order processing.
  ✗ We need absolute data consistency across microservices.

Summary #

  • At-Most-Once — The delivery guarantee where every message is processed once or not at all, minimizing duplication risk with the consequence of potential data loss.
  • Fire-and-Forget — The producer delivery pattern using the acks=0 and retries=0 properties that immediately returns success status without waiting for storage confirmation from the leader broker.
  • Commit-First Pattern — The consumer-side queue clearing method by immediately committing the last offset to the broker before executing data processing logic.
  • High-Throughput & Low-Latency — The main advantage of At-Most-Once because there’s no network overhead for acknowledgement coordination and multi-partition transaction processes.
  • Clickstream & IoT Use Cases — Perfectly suited for sensor telemetry systems, web navigation tracking, and aggregate log collection where lost data samples don’t damage global statistics.
  • Financial Transaction Danger — Using At-Most-Once semantics is strictly forbidden for high-value data like bank balance transfers or e-commerce orders because system failure risks can permanently vanish user money.
Next: At-Least-Once →
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact