EOS Limitation #

The Exactly-Once Semantics (EOS) data processing guarantee in Apache Kafka is often considered the absolute solution for all data consistency problems in distributed system architectures. This feature is indeed very powerful, but as software engineers, we must understand that EOS isn’t a limitless magic bullet (silver bullet). There are important technical and structural limitations bounding its guarantee scope. The most fundamental one is: Kafka EOS only applies to data flows entirely inside the Kafka ecosystem (Kafka-to-Kafka). When our data flow interacts with external systems — like writing to SQL/NoSQL databases, sending REST API calls to payment gateways, or updating caches in Redis — Kafka’s Exactly-Once guarantee no longer covers those systems. This article will deeply dissect Kafka EOS scope boundaries, external system integration challenges, rescue architecture solutions like the Outbox Pattern, performance overhead, and the mandatory production configuration checklist.

EOS Scope Boundaries: Only Applies to Kafka-to-Kafka Flows #

The most fundamental limitation of Kafka EOS is its internal ecosystem scope boundary (boundary of guarantee). The Kafka transaction protocol is designed to coordinate internal broker status, topics, partitions, and consumer offsets stored in the internal __consumer_offsets topic.

KAFKA EOS GUARANTEE BOUNDARY:
[Input Topic] ──> [Kafka Consumer] ──> [Application Logic] ──> [Kafka Producer] ──> [Output Topic]
└────────────────────────────── KAFKA EOS GUARANTEE ─────────────────────────────────────────┘

EXTERNAL INTEGRATION BOUNDARY (NOT Automatically Guaranteed by EOS):
[Input Topic] ──> [Kafka Consumer] ──> [Application Logic] ──> [External Database / REST API]
└─────────────── KAFKA GUARANTEE ────────────────┘ └────────── NOT EOS GUARANTEED ───────────────┘

If our application consumes messages from Kafka and writes the results to a database (for example PostgreSQL) or sends data to an external web service (for example the Stripe API), the Kafka transaction can’t guarantee that database transaction’s atomicity.

  • Why is this so? Because Kafka doesn’t support global distributed transaction protocols that block like Two-Phase Commit across external systems (for example XA Transactions). Kafka can’t send rollback commands to a PostgreSQL database or cancel Stripe HTTP API calls already processed on third-party servers if the Kafka transaction later fails.

Therefore, Kafka’s built-in exactly-once transactional processing pattern is only 100% guaranteed if we read from a Kafka topic and write back to a Kafka topic.

2-Phase Commit (2PC) vs the Kafka Transaction Protocol #

To appreciate Kafka EOS limitations, we must understand why Kafka doesn’t use the classic distributed Two-Phase Commit (2PC) protocol (like the XA/Open JTA specification) commonly used to coordinate transactions between queue systems and relational databases (for example JMS ActiveMQ with Oracle Database).

In traditional 2PC:

  1. Prepare Phase: The external transaction coordinator sends messages to the database and queue system to lock resources (blocking locks) and ask whether they’re ready to write data.
  2. Commit Phase: If all systems answer “yes”, the coordinator sends the commit command. If one answers “no” or doesn’t respond due to network disruptions, the coordinator sends rollback commands to all systems.

Why is this forbidden in the Kafka ecosystem?

  • Resource Locking Hurdle: Traditional 2PC requires databases to lock data rows (row locks) during the prepare phase until the commit finishes. If the network is slow, these locks last a long time, instantly destroying our database throughput.
  • Single Point of Failure: If the external transaction coordinator dies after the prepare phase but before the commit phase, all databases are locked in a pending state (in-doubt state), requiring manual administrator intervention to release the locks.
  • Giant Scale: Kafka is designed to process millions of messages per second. Using distributed XA locking mechanisms would make Kafka throughput plummet close to the very slow relational database throughput.

Conversely, Kafka uses a non-blocking internal 2PC variation. The coordinator broker asynchronously records status changes to the __transaction_state topic and spreads markers to regular data partition logs without ever locking partitions or forbidding consumers from reading non-transactional data. This guarantee is very fast, but the consequence is that it can’t reach and lock data rows in your external databases.

Detailed Outbox Pattern Handling #

When our microservices application must update local database status (RDBMS) while also publishing events to Kafka, the Transactional Outbox Pattern is the most reliable architecture solution for guaranteeing Exactly-Once processing semantics at the overall system level.

1. Outbox Table Schema Structure #

Our application writes to a business table (e.g., orders) and an outbox table (e.g., outbox_events) under the same local database transaction. Here’s a recommended outbox table schema structure example:

CREATE TABLE outbox_events (
    event_id UUID PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

2. CDC (Change Data Capture) Engine Workflow #

After the local database transaction is committed with ACID:

  1. A CDC engine (like Debezium installed on Kafka Connect) non-blockingly monitors the database binary log (for example the Write-Ahead Log / WAL on PostgreSQL or the Binlog on MySQL). CDC doesn’t run repeated SELECT queries burdening the database CPU.
  2. CDC automatically extracts new rows entering the outbox_events table.
  3. CDC turns those rows into Kafka messages, embeds event_id in the message header, and sends them to the destination Kafka topic (for example the orders-event-stream topic).
  4. After the message is successfully sent to Kafka, an optional background process can delete old entries in the outbox_events table so the database size doesn’t bloat.

3. CDC-to-Kafka Delivery Guarantees #

Because the CDC engine reads the database’s physical transaction log, it guarantees that every message successfully committed in the database will definitely be sent to Kafka (At-Least-Once).

  • To prevent duplication in the Kafka broker from CDC resends after connector crashes, the Kafka Connect connector is configured with producer.enable.idempotence=true.
  • Downstream consumers reading from the Kafka topic then use one of the deduplication tactics (unique constraints or Redis tables) to filter duplicates if offset commit failures happen on the consumer side itself.

External System Integration Solutions #

Although Kafka EOS doesn’t automatically cover external systems, we can use several architecture patterns to achieve Exactly-Once processing guarantees at the application level:

Tactic 1: Idempotent Deduplication on the Database Side #

This tactic leverages the idempotency capability of our target database.

  • How It Works: Ensure every message entering from Kafka has a permanent unique identifier (for example transaction_id). On the target database side, make that unique identifier the Primary Key or create a Unique Constraint status column index.
  • Write Operations: Use conditional write commands like UPSERT (Insert or Update / INSERT ... ON CONFLICT DO UPDATE in PostgreSQL, or INSERT ... ON DUPLICATE KEY UPDATE in MySQL). If a duplicate message arrives from consumer-side offset commit failures, the database only updates the old data row instead of creating a new duplicate row.

Tactic 2: The Outbox Pattern with CDC (Change Data Capture) #

If we must do local database writes and Kafka message sends simultaneously from one microservice, the best way is using the Outbox Pattern:

  1. Our application only writes business data to the main table and writes outgoing messages to a special table called the Outbox Table in one local database transaction (RDBMS ACID). This guarantees the data write and outgoing message registration are 100% atomic.
  2. A Change Data Capture component (like Debezium or a Kafka Connect Source Connector) monitors our database transaction log.
  3. Once the local database transaction is successfully committed, CDC reads the data from the Outbox Table asynchronously and sends it to the Kafka topic.
  4. Because CDC uses the At-Least-Once guarantee, messages are sent to Kafka at least once. We then enable the Idempotent Producer on the connector to ensure no duplication in the Kafka broker.
flowchart TD
    App["Client Application"] -->|"1. Local Transaction (ACID)"| DB[(RDBMS Database)]
    
    subgraph DB_Boundary["Database Boundary"]
        DB -->|"Write Business Data"| BizTable["Business Table"]
        DB -->|"Write Event Payload"| OutboxTable["Outbox Table"]
    end
    
    CDC["Debezium (CDC) / Kafka Connect"] -->|"2. Monitor Transaction Log"| OutboxTable
    CDC -->|"3. Send Events Idempotently"| Kafka["Kafka Topic"]
    
    style DB_Boundary stroke:#333,stroke-dasharray:5,5
    style DB stroke:#0288d1,stroke-width:2px
    style Kafka stroke:#2e7d32,stroke-width:2px

Performance Overhead and Impact on Throughput #

Enabling transaction and idempotence features in Apache Kafka has a direct impact on cluster performance and throughput. We must pay this durability price in the form of:

1. Increased Network Latency and CPU #

  • ACKS=ALL: Idempotence and transactions automatically force producers to use the acks=all property. This means producers must wait for messages to be copied to the entire ISR replica list before getting success confirmation. Network latency per delivery increases proportionally to the replica count and inter-rack network delay.
  • Transaction Status Logging: Every active transaction requires the coordinator to write transaction records (Ongoing, PrepareCommit, CompleteCommit) to the internal __transaction_state topic. This multiplies the write I/O volume the coordinator broker must handle.

2. Consumer-Side Latency Obstacles (LSO Blockage) #

As discussed before, consumers with the isolation.level=read_committed configuration can’t read new messages past the Last Stable Offset (LSO) boundary.

  • If a transactional producer experiences a pause or business logic error mid-way and leaves its transaction status hanging (hanging transaction) without calling commitTransaction() or abortTransaction(), that partition’s LSO freezes.
  • Downstream read_committed consumers are held (blocked) and can’t process new data from that partition, triggering a drastic consumer lag increase even though new non-transactional data is piling up on the broker.

The Danger of Long-Running Transactions #

A fatal mistake developers often make when using the Transaction API is letting transactions run too long.

  • Cause: Opening a transaction with beginTransaction(), doing complex data processing (like processing large files or waiting for slow external API calls), then finally calling commitTransaction().
  • Risks:
    1. LSO Blockage on downstream consumers lasts a long time, globally holding other systems’ data processing.
    2. Transaction Timeout: The Kafka broker has a security parameter transaction.max.timeout.ms (by default 15 minutes). If the producer transaction duration exceeds the set transaction.timeout.ms limit (default 1 minute), the broker-side transaction coordinator unilaterally aborts that transaction. Producers trying to commit after the limit passes receive TransactionTimeoutException and the entire data batch is discarded.

The Golden Rule: Open transactions as briefly as possible. Only call beginTransaction() right before calling the send() function to Kafka, and immediately call commitTransaction() right after the data batch is finished sending. Never put slow business logic processing or non-Kafka I/O calls inside an active transaction block.

Exactly-Once (EOS) Production Readiness Checklist #

Before releasing an application using Exactly-Once semantics to production, ensure all the following configuration parameters have been verified and aligned:

1. Producer Client Configuration (Producer Settings) #

  □ enable.idempotence = true (Enables per-message sequence numbers)
  □ transactional.id = [UNIQUE_STATIC_NAME] (Must be unique for each active application replica)
  □ transaction.timeout.ms = 10000 to 30000 (Set the transaction duration as short as possible)
  □ acks = all (Forced automatically by idempotence)
  □ max.in.flight.requests.per.connection = 1 to 5 (Guarantees delivery order)

2. Consumer Client Configuration (Consumer Settings) #

  □ isolation.level = read_committed (Prevents dirty reads of aborted data)
  □ enable.auto.commit = false (Offsets must be committed via sendOffsetsToTransaction)

3. Broker Cluster Configuration (Broker Settings) #

  □ transaction.state.log.replication.factor = 3 (Transaction status data resilience)
  □ transaction.state.log.min.isr = 2 (Minimum active transaction replica broker limit)
  □ transaction.max.timeout.ms = 900000 (Maximum transaction time limit tolerance on the broker)

Summary #

  • Kafka-only Boundary — The Exactly-Once (EOS) semantic guarantee in Kafka is only 100% guaranteed for internal data flows from Kafka topic to Kafka topic (Kafka-to-Kafka).
  • External Integration — Kafka transactions don’t automatically cover SQL/NoSQL database storage or third-party REST API calls, requiring additional handling tactics at the application level.
  • Idempotent Deduplication — The duplicate data handling tactic on the target database side using Primary Key/Unique Constraints and conditional write operations (UPSERT).
  • Outbox Pattern — The architecture pattern ensuring atomicity of business data writes and outgoing message registration locally in the RDBMS before being streamed idempotently to Kafka by CDC (Debezium).
  • EOS Performance Overhead — Transactions cause cluster throughput decreases and latency increases from double status log writes to __transaction_state and full replication requirements (acks=all).
  • LSO Blockage — Downstream read_committed consumers are held from reading new messages if there’s an ongoing partition transaction running too long or hanging.
  • Transaction Golden Rule — Transaction blocks must be kept as short as possible; avoid putting slow data processing or non-Kafka I/O calls between beginTransaction() and commitTransaction().
← Previous: Producer Consumer Transaction
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact