At-Least-Once vs At-Most-Once #

When designing event-driven systems using Apache Kafka, one of the most crucial aspects determining data integrity is the message delivery semantics. On the consumer side, we face two fundamental processing guarantee choices: At-Least-Once and At-Most-Once. Each choice carries very different architectural implications, forcing us to pick a trade-off between business data duplication risk or permanent data loss risk. Without deep understanding of when offsets are committed to the broker relative to business logic execution, our applications are vulnerable to hard-to-trace data inconsistencies in production.

Dissecting Processing Guarantees: The Consumer Perspective #

Fundamentally, message processing guarantees on the consumer side are determined by the chronological order of two main activities:

1. Data Processing: Running application business logic, like writing data to a database, calling external APIs, or updating memory caches.
2. Offset Commit: Reporting back to the Kafka coordinator broker the highest read position successfully processed so it’s stored in the internal __consumer_offsets topic.

Because these two activities are distributed operations separated by network I/O, we can’t execute them simultaneously in one global atomic transaction (unless we apply a special transaction pattern discussed in a separate chapter). Therefore, the execution order of these two activities produces two main semantics.

At-Most-Once #

The At-Most-Once semantic guarantees that every message sent by the broker is processed at most once by our consumer. In this semantic, there’s a possibility of messages being lost and never processed (data loss), but the system is guaranteed completely free from duplicate message processing risk (zero duplicates).

Operation Order: Commit-First #

To implement At-Most-Once, the consumer must operate with the following flow:

1. The consumer calls .poll() and obtains a batch of messages (for example, Offsets 100 to 109).
2. The consumer immediately sends an offset commit for Offset 109 to the Kafka broker. The broker records Offset 109 as the successfully read position.
3. The consumer starts processing business logic for those messages locally.

flowchart TD
    Start["1. Fetch Message Batch (Offset 100-109)"] --> Commit["2. Immediately Commit Offset 109 to Broker"]
    Commit --> Process["3. Start Local Business Logic Processing"]
    
    Process --> Success{"Was processing successful?"}
    Success -- "Yes" --> End["Done, Ready for Next Poll"]
    Success -- "No / Crash" --> Crash["Application Dies Suddenly"]
    
    Crash -. "When the Application Recovers" .-> Recover["Start Reading from Offset 110"]
    
    style Crash stroke:#c62828,stroke-width:2px
    style Recover stroke:#2e7d32,stroke-width:2px

Failure Scenario: Data Loss #

Suppose while processing data in step 3 (for example, at message Offset 105), our application container suddenly crashes from heap memory exhaustion (OOM) or a power outage.

• State on the Broker: The Kafka coordinator considers our consumer group has successfully finished reading up to Offset 109 (because the commit was sent and received by the broker in step 2).
• When the Consumer Recovers: When the application instance restarts, it queries the last read position from the broker. The broker directs the consumer to start reading from Offset 110.
• Impact: Messages from Offset 105 to 109 are lost forever from our processing system. Those messages will never be read again, leaving a data loss gap in downstream systems.

The Right Use Case for At-Most-Once #

Although data loss sounds scary, At-Most-Once is very useful for scenarios prioritizing processing speed and tolerating the loss of a few data samples, such as:

• Clickstream Collection: Analyzing user click behavior on websites. Losing one or two clicks won’t damage overall trend analysis.
• IoT Sensor Metrics: Reading temperature or humidity data from thousands of sensors every 5 seconds. If one sensor reading is lost, the next data arriving 5 seconds later replaces it.
• Log Streaming (Logstash/Splunk): Streaming server log data for general performance monitoring.

At-Least-Once #

The At-Least-Once semantic guarantees that every message sent by the broker is processed at least once by our consumer. In this semantic, not a single message is lost from our processing system (zero data loss), but there’s a risk that one or several messages are processed more than once (duplication).

Operation Order: Process-First #

To implement At-Least-Once, the consumer must reverse its operation order:

1. The consumer calls .poll() and obtains a batch of messages (for example, Offsets 100 to 109).
2. The consumer runs all business logic for those messages until complete and successfully stored in the destination database.
3. After all processing is confirmed successful without errors, the consumer then sends an offset commit for Offset 109 to the Kafka broker.

flowchart TD
    Start["1. Fetch Message Batch (Offset 100-109)"] --> Process["2. Run Business Logic & Save to DB"]
    Process --> Success{"Was processing successful?"}
    
    Success -- "Yes" --> Commit["3. Commit Offset 109 to Broker"]
    Success -- "No / Crash" --> Crash["Application Dies Suddenly"]
    
    Commit --> End["Done, Ready for Next Poll"]
    
    Crash -. "When the Application Recovers" .-> Recover["Re-read from Offset 100"]
    
    style Crash stroke:#c62828,stroke-width:2px
    style Recover stroke:#2e7d32,stroke-width:2px

Failure Scenario: Data Duplication #

Suppose after step 2 finishes (all data up to Offset 109 was successfully written to the RDBMS database), the network drops before step 3 executes. The consumer fails to send the offset commit signal to the coordinator broker.

• State on the Broker: The Kafka coordinator records our last commit offset still at 99.
• When the Consumer Recovers (or a Rebalance Happens): The new consumer assigned to take over that partition queries the last position from the broker. The broker directs the consumer to re-read from Offset 100.
• Impact: The consumer reprocesses messages from Offset 100 to 109. Because that data was already stored in the database in the first attempt, this reprocessing triggers duplicate data entries (for example, a payment transaction debited twice) if we don’t implement an idempotency mechanism in the database.

The Effect of Local Retry Mechanisms on Consumer Lag #

When using At-Least-Once, a business logic failure on one message holds the offset commit for the entire batch of messages being consumed.

• If we apply a local retry mechanism with a backoff retry strategy, the main consumer thread blocks the next .poll() call until the retry process finishes or the message is routed to a Dead Letter Queue.
• During this retry pause, the committed offset on the broker doesn’t advance. The Consumer Lag metric indicator on our monitoring dashboard detects a sharp lag spike. This is normal because we prioritize data processing accuracy over data flow speed.

Bridging Toward Exactly-Once Semantics (EOS) #

For developers wanting distributed system perfection, the ideal choice is Exactly-Once Semantics (EOS) — the guarantee where data is processed exactly once (no data loss and no duplication).

• Within the Kafka ecosystem, pure Exactly-Once is achieved by integrating transactional producers and transactional consumers.
• On transactional consumers, the isolation.level configuration property plays an important role:
  • read_uncommitted (Default): Consumers immediately read all messages written to the broker, including messages from transactional producers whose status isn’t committed yet (still pending) or even aborted/cancelled transaction messages.
  • read_committed: Consumers only read non-transactional messages or transactional messages successfully committed by producers. Messages from aborted transactions are automatically skipped by consumers.
• Deep details of this two-phase commit transaction flow will be comprehensively discussed in the next Delivery Semantics article series.

Processing Guarantee Comparison Table #

The table below summarizes the strategic trade-offs between the two data delivery semantic types:

Code Implementation: At-Most-Once vs At-Least-Once #

Let’s see concretely how code writing affects the resulting processing semantics.

Java SDK: At-Most-Once Implementation Example (Commit-First) #

To achieve At-Most-Once semantics instantly, we can use the built-in Auto-Commit with minimal wait time, or call commitSync() right after data is fetched from the .poll() function.

// AT-MOST-ONCE EXAMPLE: Prioritizing duplication prevention over data loss
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "iot-sensor-consumer");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // Disable auto for manual control
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("sensor-telemetry"));

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        
        if (!records.isEmpty()) {
            // ✓ COMMIT FIRST: Immediately report the successful read position to the Kafka broker
            // before the crash-prone business logic runs.
            consumer.commitSync(); 
            
            for (ConsumerRecord<String, String> record : records) {
                // If this line crashes or throws an exception, messages in the batch
                // are skipped forever because the broker already recorded the latest commit offset.
                saveSensorReadingToMemoryCache(record.value());
            }
        }
    }
} finally {
    consumer.close();
}

Java SDK: At-Least-Once Implementation Example (Process-First) #

To guarantee no data is lost, we must disable auto-commit (enable.auto.commit=false), run business logic to completion inside a try-catch block, then do a manual commit only after confirmed free from error exceptions.

// AT-LEAST-ONCE EXAMPLE: Guaranteeing no data loss (Main Recommendation)
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing-consumer");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // ✓ Must be false
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singletonList("billing-events"));

try {
    while (true) {
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
        
        for (ConsumerRecord<String, String> record : records) {
            try {
                // 1. Run the important business process (write to the transaction database)
                executeBillingTransactionInDatabase(record.value());
            } catch (Exception e) {
                // If the database write fails, stop batch processing immediately.
                // Don't call commitSync(). When the loop cycles back or crashes,
                // this data is re-fetched from the broker to retry.
                log.error("Database processing failed for offset: " + record.offset(), e);
                throw new ProcessingFailedException("Failed to write to DB", e);
            }
        }
        
        // 2. ✓ COMMIT LAST: Only commit offsets after all records
        // in the batch are successfully stored in the database without errors.
        if (!records.isEmpty()) {
            consumer.commitSync();
        }
    }
} catch (ProcessingFailedException pfe) {
    // Recovery error handling (e.g., pause before retry)
    sleepAndRecover();
} finally {
    consumer.close();
}

Why Does At-Least-Once Require Idempotency? #

If we choose the At-Least-Once guarantee (which is the standard for commercial business applications), we must accept the distributed mathematics reality: message duplication will definitely happen sooner or later. The internet is never perfect. Connections dropping right after database writes complete is a common occurrence on production servers.

Therefore, if we use At-Least-Once semantics, our consumer applications must implement an Idempotent Consumer mechanism.

• Idempotency means running an operation multiple times with the same parameters produces the same system effect as running it the first time.
• Example RDBMS tactic using a deduplication table:

-- Deduplication Table Schema for filtering unique Event IDs
CREATE TABLE processed_events (
    event_id VARCHAR(255) PRIMARY KEY,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

// EXAMPLE OF IDEMPOTENCY HANDLING IN CONSUMER CODE
try (Connection conn = dataSource.getConnection()) {
    conn.setAutoCommit(false);
    
    // ✓ Try recording the Event ID to the database first.
    // If the Event ID has already been processed, the Primary Key constraint
    // throws a DuplicateKeyException, cutting off duplicate transaction processing.
    try (PreparedStatement stmt = conn.prepareStatement(
        "INSERT INTO processed_events (event_id) VALUES (?)"
    )) {
        stmt.setString(1, record.key());
        stmt.executeUpdate();
    }
    
    // Continue the business process
    updateAccountBalance(record.value());
    conn.commit();
} catch (SQLIntegrityConstraintViolationException e) {
    // Message detected as duplicate, safely ignore without reprocessing business
    log.warn("Detected duplicate message with ID: {}, skipping the process.", record.key());
}

Summary #

• At-Most-Once — The processing guarantee where every message is processed at most once. Eliminates duplication risk but opens a data loss gap.
• Commit-First Pattern — The At-Most-Once approach is achieved by committing offsets to the broker right after messages are received from .poll(), before business logic runs.
• At-Least-Once — The processing guarantee where messages are guaranteed not to be lost from the system flow, but ready to accept message duplication risk.
• Commit-Last Pattern — The At-Least-Once approach is achieved by disabling auto-commit and only calling commitSync() after all business logic is confirmed successfully stored in the database.
• Important Trade-Off — Choosing a semantic guarantee is a business trade-off: decide whether our application tolerates losing small data samples (At-Most-Once) or temporary data duplication (At-Least-Once).
• Idempotency Mandate — Using At-Least-Once requires our application architecture to have an idempotency handling layer on downstream database storage to dampen duplicate processing impacts.

← Previous: Rebalance Process Next: Duplicate Message →