What is Kafka Streams? #

In the modern event-driven architecture era, processing speed is the main differentiator between responsive systems and sluggish ones. Traditional data processing relying on batch models (like scheduled nightly data processing using Hadoop or Spark) is no longer adequate for use cases demanding instant reactions, like bank fraud transaction detection, dynamic e-commerce recommendations, or server fleet health monitoring. To bridge this need, Apache Kafka provides a stream processing library called Kafka Streams. Through this article, we’ll deeply dissect Kafka Streams’ basic philosophy, why it’s designed as an embedded library, how its internal threading model architecture operates, and a comprehensive comparison with big processing frameworks like Apache Spark and Apache Flink.

Embedded Library vs Centralized Framework Philosophy #

When we talk about stream processing, old paradigms often steer us toward thinking of large, complex distributed compute clusters like Apache Spark Streaming, Apache Flink, or Apache Storm. Those frameworks are designed with centralized compute models requiring their own master-worker clusters to run. On the other hand, Kafka Streams takes a completely opposite design direction.

1. Simple Java Library Model (Embedded Library) #

Kafka Streams isn’t a data processing system with a standalone execution engine. It’s a regular Java client library (kafka-streams.jar) we import into our application code.

• No Additional Clusters: We don’t need to rent, configure, monitor, and manage additional server clusters like Mesos, YARN, or Spark/Flink Standalone Managers. Our application runs as a standard operating system process.
• Deployment Compatibility: Because applications using Kafka Streams are just regular Java applications (whether packaged as standalone JARs, Spring Boot applications, or Quarkus), we can deploy them any way we deploy other microservices. We can wrap them in Docker containers and deploy to Kubernetes, Nomad, AWS EC2 VMs, or even bare-metal.

2. Symbiotic Relationship with Apache Kafka #

Kafka Streams is specially designed from the ground up to work exclusively with Apache Kafka. It leverages all Kafka’s built-in capabilities to solve complex distributed system challenges.

• State Management: Stores local state using RocksDB and replicates it to Kafka brokers through changelog topics.
• Fault Tolerance: Relies on Kafka partition architecture and consumer group rebalance mechanisms to redistribute workloads if application instances fail.
• Scalability: Horizontal scalability is achieved simply by running new application instances with the same application.id configuration. Kafka automatically divides input partitions to the newly started instance.

flowchart TD
    subgraph SPARK["Spark / Flink Model (Heavy Cluster)"]
        direction TB
        Master["Spark Master Node"]
        Worker1["Worker Node 1"]
        Worker2["Worker Node 2"]
        Master -. Manages tasks .-> Worker1
        Master -. Manages tasks .-> Worker2
        DataIn["Kafka Topic"] --> Master
        Worker1 --> DataOut["Result Topic"]
        Worker2 --> DataOut
    end

    subgraph KSTREAMS["Kafka Streams Model (Embedded Library)"]
        direction TB
        App1["Client Application (Spring Boot + KStreams)"]
        App2["Client Application (Spring Boot + KStreams)"]
        Broker["Kafka Cluster (Broker)"]
        
        Broker -- "Reads & Writes Events directly" --> App1
        Broker -- "Reads & Writes Events directly" --> App2
    end
    
    style Master stroke:#d32f2f,stroke-width:2px
    style Worker1 stroke:#d32f2f,stroke-width:2px
    style Worker2 stroke:#d32f2f,stroke-width:2px
    style App1 stroke:#388e3c,stroke-width:2px
    style App2 stroke:#388e3c,stroke-width:2px
    style Broker stroke:#0288d1,stroke-width:2px

Internal Architecture: Threading Model and Task Assignment #

Behind its deployment simplicity, Kafka Streams holds a very structured partition handling and threading architecture. Understanding these components is crucial so we can tune application performance precisely in production.

1. Partition-to-Task Relationship (Task Assignment) #

Kafka Streams breaks our processing logic topology into one or several parallel work units called StreamTasks. The StreamTask count is directly determined by the maximum partition count of the input topics our application reads.

For example, if our application reads from Topic_A (which has 6 partitions) and writes to Topic_B, Kafka Streams creates exactly 6 StreamTasks (Task 0 to Task 5).

• Data Isolation: Each Task exclusively processes data from one specific input partition. Data is read from the partition, processed through the logic operator chain, and sent to the output partition.
• State Store: If our process is stateful (like per-minute data aggregation), each Task has its own local State Store (RocksDB) containing data only from the partition assigned to it. This prevents cross-thread memory conflicts because there’s no shared-state in memory globally.

2. Processing Threads (StreamThread) #

The StreamThread is the actual Java thread executing StreamTask logic. We can set the number of StreamThreads per application instance using the num.stream.threads configuration parameter (default is 1).

THREAD AND TASK RELATIONSHIP MAP:
-------------------------------------------------------------------
Application Instance 1 (num.stream.threads = 2):
  ├── StreamThread 1 (Executes Task 0, Task 1, Task 2)
  └── StreamThread 2 (Executes Task 3, Task 4, Task 5)
-------------------------------------------------------------------

If one StreamThread dies from an unhandled exception, Kafka Streams detects that death and automatically moves the hanging StreamTasks to other living threads in that instance or to other application instances in the cluster through group rebalance processes.

3. Dynamic Horizontal Scalability #

Let’s review a horizontal scalability scenario where we have an input topic with 4 partitions, and we start our application at various instance density levels:

• Scenario 1 (Single Instance): We start 1 application instance with num.stream.threads=2. That instance has 2 active threads, where each thread executes 2 StreamTasks in parallel.
• Scenario 2 (Two Instances): We start a second instance with the same application.id configuration. The Kafka rebalance protocol dynamically triggers. Instance 1 now processes Task 0 and Task 1, while Instance 2 processes Task 2 and Task 3. This task migration process runs transparently in the background without losing local state thanks to changelog replication.
• Scenario 3 (Excess Instances): We start 5 application instances for an input topic with 4 total partitions. Because the partition count is only 4, only 4 StreamTasks can form. The 5th instance runs in idle (standby) status without processing any data, ready to instantly take over tasks if one of the 4 active instances crashes.

Implementation Code: Anti-Pattern vs the Kafka Streams Solution #

To truly understand why we should use Kafka Streams rather than trying to build our own stream processing engine using regular Kafka Consumers, let’s compare both approaches through real implementation examples below.

Use Case Problem #

Imagine we want to read shopping transaction event streams (purchase-events) sent in JSON format, filter transactions valued below $10, and write the buyer data along with their purchase values to a destination topic (large-purchases).

Anti-Pattern: Using a Manual Consumer Loop Pattern #

Below is Java code attempting to achieve that goal manually using a regular consumer loop.

// ANTI-PATTERN: Building manual stream processing logic with a regular Consumer Loop.
// ✗ Very hard to manage fault tolerance, scaling, state stores, and offset commit handling.
public class ManualStreamProcessor {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "manual-processor-group");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        consumer.subscribe(Collections.singletonList("purchase-events"));

        Properties prodProps = new Properties();
        prodProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        prodProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        prodProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        KafkaProducer<String, String> producer = new KafkaProducer<>(prodProps);

        try {
            while (true) {
                // ✗ Blocking poll loop calls that kill rebalance performance if processed slowly
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
                for (ConsumerRecord<String, String> record : records) {
                    String purchaseJson = record.value();
                    double amount = parseAmount(purchaseJson);

                    // Doing manual filtering
                    if (amount >= 10.0) {
                        ProducerRecord<String, String> outRecord = 
                            new ProducerRecord<>("large-purchases", record.key(), purchaseJson);
                        
                        // ✗ Async sending without atomic transaction coordination (potential data loss/duplication)
                        producer.send(outRecord);
                    }
                }
                // ✗ Manual offset commits potentially trigger double At-Least-Once if a crash happens before the commit succeeds
                consumer.commitSync();
            }
        } finally {
            consumer.close();
            producer.close();
        }
    }

    private static double parseAmount(String json) {
        // Simple parsing logic
        return 15.0; // Example simulated purchase value
    }
}

Why Is the Anti-Pattern Code Above Very Dangerous? #

1. No State Management: If we want to do aggregation (e.g., calculating total purchases per user over the last 5 minutes), we must create an external memory store (like Redis) manually. This adds new infrastructure dependencies and slows latency because we must access cross-node networks.
2. Potential Data Duplication: There’s no atomic transactional processing guarantee between producer writes and consumer offset commits.
3. Complex Threading: To add performance, we must manually write Java multi-thread management logic on top of the Consumer API, which is prone to race condition errors or memory leaks.

Practical Solution: Using the Kafka Streams DSL API #

Below is a standard implementation using the Kafka Streams DSL API. This code is very concise, safe, automatically managed by the internal system, and ready to scale.

// CORRECT: Using the Kafka Streams DSL to define stream processing topologies.
// ✓ Far safer, automatically manages RocksDB state stores, threading, and fault-tolerance.
public class KafkaStreamsDslProcessor {
    public static void main(String[] args) {
        Properties config = new Properties();
        
        // ✓ The Application ID acts as the consumer Group ID and namespace for local state stores
        config.put(StreamsConfig.APPLICATION_ID_CONFIG, "purchase-filtering-service");
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        
        // Default serializer / deserializer data type configuration
        config.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        config.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
        
        // Optimizing internal processing
        config.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2); // Using 2 parallel threads

        // Creating the stream processing topology builder
        StreamsBuilder builder = new StreamsBuilder();

        // ✓ Reading the input stream from the source topic non-blockingly
        KStream<String, String> purchases = builder.stream("purchase-events");

        // Defining the asynchronous filter topology using lambda expressions
        purchases
            .filter((key, value) -> parseAmount(value) >= 10.0)
            // ✓ Writing processing results directly to the destination topic
            .to("large-purchases");

        // Building the physical logic topology
        Topology topology = builder.build();
        
        // Initializing the Kafka Streams instance
        KafkaStreams streams = new KafkaStreams(topology, config);

        // Adding a shutdown hook so the application closes connections cleanly when shut down
        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));

        log.info("Starting stream topology processing...");
        streams.start();
    }

    private static double parseAmount(String json) {
        // JSON parsing logic, e.g., using Jackson
        return 20.0;
    }
}

Comparison Matrix: Kafka Streams vs Spark vs Flink #

So we don’t choose the wrong technology platform when designing systems, we need to map Kafka Streams’ characteristics against the two other giant stream processing frameworks in the industry:

When to Choose Kafka Streams? #

To ease architecture decision-making in our team, here’s a simple decision flow chart for choosing among the available options:

CHOOSE KAFKA STREAMS IF:
  ✓ Our main input data source and output destination are Apache Kafka.
  ✓ We want to deploy our data processing application to Kubernetes/Docker like other microservices.
  ✓ Our developer team is very familiar with the Java/JVM ecosystem.
  ✓ We want to avoid the operational cost and maintenance overhead of new external clusters.

CHOOSE APACHE SPARK / FLINK IF:
  ✗ We need to combine Kafka data directly with other external data sources like Hadoop HDFS or RDBMS in large quantities.
  ✗ We need advanced Complex Event Processing (CEP) features not supported by the basic DSL.
  ✗ We're doing giant-scale batch analytics that aren't based on continuous event streams.

Summary #

• Kafka Streams — An embedded library Java client used for processing data to and from Apache Kafka in real-time without needing additional compute clusters.
• Lightweight Philosophy — No special server installation needed; applications using Kafka Streams can be deployed like regular microservices using Docker or Kubernetes.
• StreamTask — The smallest parallel work unit in Kafka Streams, where the count is dynamically determined by the maximum partition count of input topics.
• StreamThread — The actual Java data processing thread responsible for executing computation logic from one or several StreamTasks in parallel.
• RocksDB — A local key-value database engine outside JVM heap memory (off-heap) used to efficiently store stateful data for each partition.
• Embedded vs Cluster — Kafka Streams prioritizes operational simplicity and local integration performance, unlike Spark/Flink which prioritize massive-scale computing across various storage media types.