Stream vs Table #

In the world of real-time data processing, our biggest challenge is often aligning two structurally different types of information: the continuously flowing transaction history and the current status of our business entities. For example, in banking systems, we have a flow of money transfer transaction history (happening every second) and we also have a table of users’ current account balances. Apache Kafka solves this challenge by introducing the Stream-Table Duality concept, realized through the programming abstractions KStream, KTable, and GlobalKTable. Through this article, we’ll deeply dissect the fundamental differences between data streams and tables, how each abstraction works in Kafka Streams, specific use cases for all three, and how to safely transform data between those types without triggering memory leaks or partition mismatches.

The Fundamental Stream-Table Duality Concept #

Before we dive into code and internal architecture, we must understand this very elegant theoretical foundation: a data stream can be turned into a table, and a table can be turned back into a data stream.

1. A Close Two-Way Relationship #

This duality works in two complementary directions forming the basis of stateful computation:

• Stream to Table (Aggregation): Data streams represent history from the past to the present. Every new record is an independent event. If we replay all events in a data stream from start to finish and aggregate them by specific keys, we produce a state table. This is similar to how an accounting cash book (debit/credit transaction stream) produces a final ledger balance (table). Without the original data stream, that state table can’t be built from scratch.
• Table to Stream (Changelog): A table represents the latest state. If we monitor every modification (additions, changes, deletions) on a table and emit those changes as new events, we produce a changelog stream of change logs. In traditional databases, this is equivalent to the Transaction Log (like the WAL in PostgreSQL or the Binlog in MySQL) emitting row changes to replication or CDC (Change Data Capture) systems.

2. Logical Implementation in Kafka #

Inside Kafka, this duality is very real. A Kafka topic is physically an event log (stream). However, by using Log Compaction configuration (cleanup.policy=compact), Kafka retains only the latest record for each key. When a Kafka Streams application reads that compacted topic as a KTable, it’s reconstructing that state table into the local RocksDB database from the original stream log stored on brokers.

flowchart LR
    subgraph STREAMS["KStream (Append-Only Events)"]
        direction LR
        E1["UserA: +$100"] --> E2["UserA: -$30"]
        E2 --> E3["UserB: +$50"]
        E3 --> E4["UserA: +$10"]
    end

    subgraph DUALITY["Duality / Transformation"]
        direction TB
        Agg["Aggregation (Replay & GroupBy Key)"]
        Change["Changelog (Emit State Changes)"]
    end

    subgraph TABLES["KTable (Upsert State Table)"]
        direction TB
        State1["UserA: Balance $80"]
        State2["UserB: Balance $50"]
    end

    STREAMS -->|"GroupBy & Aggregate"| Agg
    Agg --> TABLES
    TABLES -->|"Emit Changes"| Change
    Change --> STREAMS

    style E1 stroke:#0288d1,stroke-width:2px
    style E2 stroke:#0288d1,stroke-width:2px
    style E3 stroke:#0288d1,stroke-width:2px
    style E4 stroke:#0288d1,stroke-width:2px
    style State1 stroke:#388e3c,stroke-width:2px
    style State2 stroke:#388e3c,stroke-width:2px

KStream: Append-Only Event Streams #

KStream is the abstraction of ordinary event data streams. Every data record entering a KStream is treated as an independent, append-only entity (only added at the end of the log).

Main KStream Characteristics #

• Fact Semantics: Data in a KStream is immutable. Once an event happens and is recorded to the Kafka broker, that event is historical fact. We can’t delete or modify that record. All we can do is send new events to cancel or revise the old event’s impact.
• Duplicate Key Handling: If two data records with the same key enter a KStream, both are considered two different, independent events. Both are processed one by one sequentially. For example, if a user presses the “Buy” button twice, the KStream records two separate transactions, not updating the first transaction.
• Use Case Examples:
  • GPS coordinate maps from delivery fleets sent every 5 seconds.
  • User clickstreams on e-commerce websites for conversion funnel analysis.
  • Web server access logs for real-time security monitoring.
  • Factory machine temperature sensor signals for predictive maintenance.

KTable: Stateful Changelog Tables #

KTable is the abstraction of local state tables. Unlike KStreams, every data record entering a KTable is treated as an upsert (update if it exists, insert if it doesn’t) based on its key.

Main KTable Characteristics #

• State Semantics: Data in a KTable represents the last status of an entity at a specific point in time. We use KTables when we only care about the latest value of a key.
• Duplicate Key Handling: If a new record arrives with a key already in the KTable, the old record is overwritten with the new value in local memory. RocksDB, the local storage engine of KTables, instantly updates that key-value entry.
• Tombstone (Deletion): KTables support explicit data deletion. If we send a record with a specific key but a null value (called a tombstone marker), the KTable removes that key from its local table and emits that deletion event to downstream systems.
• Use Case Examples:
  • Running account balances of bank customers.
  • Courier activity status (whether online, busy, or offline).
  • Detailed e-commerce customer profiles (like current shipping addresses or membership types).
  • Latest stock prices on the stock exchange.

GlobalKTable: Globally Replicated Tables #

GlobalKTable is similar to a regular KTable because both use upsert semantics to store the latest state. However, their fundamental difference lies in how data is distributed across our application instances and how memory consumption is managed.

KTable vs GlobalKTable Architecture Differences #

• KTable (Distributed Partitions): KTables are locally partitioned according to the Kafka topic partition division. If the input topic has 4 partitions and we have 4 StreamTasks running on 2 different application instances, each instance only loads data from 2 partitions (50% of total data).
  • Advantages: Excellent horizontal scalability. If data volume swells, we just add new application instances to split the RocksDB memory load.
  • Disadvantages: Requires strict co-partitioning rules if we want to join a KStream with a KTable. Transaction keys in the KStream and profile keys in the KTable must be partitioned the same way on brokers.
• GlobalKTable (Full Replication): GlobalKTables load 100% of the data from the input topic into every running application instance, regardless of how many partitions the input topic has and how many tasks are assigned to that instance.
  • Advantages: Greatly simplifies the join process. We can join any KStream with a GlobalKTable without re-partitioning, because the lookup data is guaranteed available in the instance’s local memory at that moment.
  • Disadvantages: Consumes far more RAM (JVM heap) memory resources and disk space (RocksDB) because of full duplication. If the data in the GlobalKTable topic exceeds one VM server’s memory capacity, the application experiences OutOfMemoryError. Therefore, GlobalKTables are only suitable for small static reference tables (like postal code tables, product category lists, or currency tables).

Java Implementation Code: KStream, KTable, and GlobalKTable #

Let’s look at Java code examples for initializing these three abstraction types using the Kafka Streams DSL API.

Implementation Scenario #

We want to build an e-commerce payment transaction processing system. We have:

1. The payment-transactions topic containing transaction streams (KStream).
2. The user-profiles topic containing user profiles and membership levels that can change anytime (KTable).
3. The currency-rates topic containing static foreign currency exchange rates (GlobalKTable).

// CORRECT: Initializing and connecting KStream, KTable, and GlobalKTable in the Java SDK.
// ✓ Complying with co-partitioning rules for local KTables and full replication for GlobalKTables.
public class StreamTableDualityExample {
    public static void main(String[] args) {
        StreamsBuilder builder = new StreamsBuilder();

        // 1. Initializing the KStream (Append-Only Log)
        // ✓ Reading every transaction as an independent event
        KStream<String, String> transactionStream = builder.stream(
            "payment-transactions",
            Consumed.with(Serdes.String(), Serdes.String())
        );

        // 2. Initializing the KTable (State Table - Local Partitions)
        // ✓ Automatically stores the latest user profile state per local partition
        // Local RocksDB stores this data and updates key-values if new profiles arrive
        KTable<String, String> userProfileTable = builder.table(
            "user-profiles",
            Consumed.with(Serdes.String(), Serdes.String())
        );

        // 3. Initializing the GlobalKTable (Globally Replicated Table)
        // ✓ All currency rate data is replicated to every instance regardless of partitions
        GlobalKTable<String, String> currencyRateTable = builder.globalTable(
            "currency-rates",
            Consumed.with(Serdes.String(), Serdes.String())
        );

        // Example Business Logic: Simple Transaction Filtering
        transactionStream
            .filter((key, value) -> value != null && value.contains("SUCCESS"))
            .to("successful-payments");

        Topology topology = builder.build();
    }
}

Transformation: Converting KStream to KTable (and Vice Versa) #

One of Kafka Streams’ main strengths is its flexibility to dynamically transition between both models.

Anti-Pattern: Using Manual Caches or External Threads #

Trying to aggregate a KStream into a status table using global Java variables (static HashMap) or manual scheduler threads is a fatal mistake often made by beginner developers.

// ANTI-PATTERN: Doing stateful KStream-to-KTable calculations manually with a local HashMap.
// ✗ State is lost when the application instance crashes, triggers memory inconsistencies, and violates thread parallelism.
public class ManualStateAggregation {
    // ✗ DANGER: A static HashMap isn't thread-safe and isn't backed up to the broker (disaster recovery fails!)
    private static final Map<String, Double> userBalances = new ConcurrentHashMap<>();

    public static void processStreamManual(KStream<String, String> stream) {
        stream.foreach((userId, transactionJson) -> {
            double amount = parseAmount(transactionJson);
            
            // ✗ Manual internal memory modification without transactional RocksDB state store coordination
            userBalances.merge(userId, amount, Double::sum);
            
            log.info("User {} balance updated to {}", userId, userBalances.get(userId));
        });
    }

    private static double parseAmount(String json) { return 100.0; }
}

Why Is the Manual Pattern Above Very Dangerous? #

1. Data Loss on Crash: A static HashMap only lives in the JVM instance’s RAM memory. If Kubernetes restarts our application container because of a liveness probe failure, all that running balance data is permanently lost.
2. Race Conditions: When we increase the thread count via the num.stream.threads configuration, multiple data processing threads access the same HashMap in parallel, triggering memory contention.
3. Scaling Inability: If we deploy a second instance of this microservice, the new instance has no access to the HashMap in the first instance. Balance data is split without synchronization.

Practical Solution: Using the Stateful Aggregation API #

Kafka Streams provides managed DSL operators to safely reduce or aggregate KStreams into KTables. The state store is automatically backed by local RocksDB and a Changelog topic under the hood.

// CORRECT: Converting a KStream into a KTable through GROUP BY and AGGREGATE operations.
// ✓ Safe from crashes, integrated with RocksDB, and fully backed up to the internal changelog topic.
public class StreamToTableDuality {
    public static KTable<String, Double> aggregateStream(KStream<String, String> transactionStream) {
        
        return transactionStream
            // Step A: Group the data stream by User ID (Key)
            // ✓ This operation triggers internal repartition if the key changes, ensuring co-partitioning.
            .groupByKey(Grouped.with(Serdes.String(), Serdes.String()))
            
            // Step B: Aggregate transaction values into the local State Store
            .aggregate(
                // 1. Initializer: Set the initial balance to 0.0
                () -> 0.0,
                
                // 2. Aggregator: Add the new transaction amount to the running balance
                (userId, transactionJson, currentBalance) -> {
                    double amount = parseAmount(transactionJson);
                    return currentBalance + amount;
                },
                
                // 3. Materialized: Configure the RocksDB state store named "user-balance-store"
                // Local RocksDB stores the state, and Kafka Streams asynchronously backs up
                // every change to the internal changelog topic named: application-id-user-balance-store-changelog
                Materialized.<String, Double, KeyValueStore<Bytes, byte[]>>as("user-balance-store")
                    .withKeySerde(Serdes.String())
                    .withValueSerde(Serdes.Double())
            ); // ✓ The output of aggregate() is a KTable<String, Double>
    }

    private static double parseAmount(String json) {
        // Parsing logic
        return 10.0;
    }
}

If we want to do the reverse (converting a KTable back into a KStream), we just call the toStream() function on our KTable object:

KTable<String, Double> userBalances = aggregateStream(transactionStream);

// Converting the running balance KTable into a KStream (Changelog stream) to send to an external topic
KStream<String, Double> balanceChangelogStream = userBalances.toStream();
balanceChangelogStream.to("user-balance-updates", Produced.with(Serdes.String(), Serdes.Double()));

By calling toStream(), every time a user’s balance is updated in RocksDB, the KTable emits one change record containing the User ID and New Balance to the destination user-balance-updates topic.

Common Problems and How to Solve Them (Troubleshooting) #

When using KTables and GlobalKTables in production, there are several common obstacles we must anticipate preventively:

1. Memory Leaks (Out of Memory) on GlobalKTables #

• Problem: The topic read as a GlobalKTable keeps growing data over time without limits, causing JVM heap memory on every instance to run out.
• Solution: Make sure the underlying Kafka topic of the GlobalKTable is configured with a compressed cleanup policy (cleanup.policy=compact) and strict segment size retention limits. Always monitor JVM memory usage metrics on our JVM dashboards.

2. RocksDB Sync Gaps #

• Problem: When the application restarts on a new Kubernetes node, RocksDB takes a long time to replay data from the Kafka changelog topic before it’s ready to serve transactions (cold start latency).
• Solution: Enable the Standby Replicas feature by setting the num.standby.replicas=1 configuration. This instructs other instances to passively duplicate the changelog so they’re ready to instantly take over task ownership without a long restore process.

Comparison Matrix: KStream vs KTable vs GlobalKTable #

To ease data type selection when designing architectures, here’s a comparison summary of the three:

Summary #

• Stream-Table Duality — A two-way relationship where event streams (KStream) can be accumulated into status tables (KTable), and status tables can be monitored for changes back into event streams.
• KStream — The append-only data abstraction where every new record is considered a historical fact that doesn’t modify previous records.
• KTable — The stateful data abstraction where new records with the same key overwrite old records (upsert), and support deletion via null records (tombstones).
• GlobalKTable — A data table abstraction where 100% of the data is replicated to all application instances, freeing join requirements from co-partitioning conditions at the cost of high memory consumption.
• Tombstone Marker — An event with a null value used in KTables/GlobalKTables to permanently delete the related key from the RocksDB local state store.
• toStream() — An instant method to expose state transitions from a KTable back into a changelog event stream (KStream) so it can be sent to other Kafka topics.

← Previous: What is Kafka Streams? Next: Topology vs Processor →