What is Kafka? #

In today’s data-driven application landscape, the amount of information flowing through a digital system grows exponentially. The monolithic systems of the past, which relied on a single central relational database to handle all recording needs, have shifted toward distributed microservices architectures. In this new environment, dozens or even hundreds of independent services must constantly exchange information quickly and reliably. We need an intermediary system that doesn’t just act as an ordinary data delivery bridge, but is also able to hold massive volumes of data safely. This is where Apache Kafka steps in as the distributed event streaming platform that has become a global industry standard.

In this article, we’ll take a thorough look at what Apache Kafka is, the story behind its creation, its conceptual definition as an event streaming platform, its core architecture, the design characteristics that make it so fast, and the various real-world use cases across the industry. With this foundation in place, you’ll have solid footing before moving on to deeper technical configuration in the modules that follow.

The Birth of Apache Kafka #

To truly understand why Apache Kafka is designed the way it is, we need to look back at its origins in 2010. At the time, the engineering team at LinkedIn was facing a major challenge around their internal data integration. LinkedIn had a very complex architecture with many custom data pipelines connecting application systems to analytical databases, performance monitoring systems, and search engines.

Jay Kreps, Neha Narkhede, and Jun Rao were the lead engineers who kickstarted this project. They realized that those point-to-point custom data pipelines were extremely hard to maintain. Every time a new service was added, the team had to write new integration code. The traditional message queue systems available at the time (such as JMS or RabbitMQ) couldn’t handle LinkedIn’s massive user activity tracking load, which generated billions of events per day.

flowchart TD
    subgraph P2P["Point-to-Point Architecture at LinkedIn (Before Kafka)"]
        direction TB
        Web["Web Application"]
        Web -->|Custom API| DB["Analytics Database"]
        Web -->|Custom Log| Mon["Monitoring System"]
        Web -->|Custom Sync| Search["Search Index (Lucene)"]
    end

They decided to build a brand-new system from scratch that adopted the concept of a relational database’s transaction log (commit log), but implemented it as a large-scale distributed system. Jay Kreps named the system Kafka, inspired by the famous author Franz Kafka, because he liked his works and realized that Kafka was designed as a system highly optimized for writing activity.

In 2011, LinkedIn donated the project to the Apache Software Foundation as an open-source project. Kafka quickly caught the attention of the global developer community and soon graduated to become an Apache Top-Level Project in 2012. A few years later, in 2014, the three Kafka founders left LinkedIn to found Confluent, a company focused on providing an enterprise-grade Kafka platform and driving the Kafka ecosystem into the giant it is today.

Definition: More Than Just a Message Broker #

Many people learning Kafka for the first time often equate it with traditional message brokers like RabbitMQ, ActiveMQ, or Amazon SQS. Although Kafka can be used for messaging, equating Kafka with a traditional message broker is a major mistake.

Apache Kafka is officially defined as a distributed Event Streaming Platform. This fundamental difference lies in three core capabilities that Kafka combines in a single platform:

1. Publish and Subscribe: Sending and receiving streams of events, similar to the function of a traditional message queue.
2. Store: Storing event streams safely, persistently, and durably in a distributed storage cluster for as long as you decide.
3. Process: Processing and reacting to event streams in real-time, right as events happen.

flowchart TD
    subgraph APACHE_KAFKA["APACHE KAFKA PLATFORM"]
        direction TB
        PUB_SUB["Publish/Subscribe <br/> Send & Receive Events"]
        STORAGE["Storage <br/> Store Events Persistently"]
        PROCESSING["Processing <br/> Process Event Streams in Real-Time"]
    end

In a traditional queue system, once a message is read by a consumer, it is usually deleted from the system immediately. In Kafka, however, data is stored persistently on disk and is not deleted right after being read. The data stays there and can be read repeatedly by different consumers, or even re-read by the same consumer from the beginning of history (replayability). This characteristic is what radically sets Kafka apart from ordinary messaging technology.

Apache Kafka’s Core Architecture #

To understand how Kafka manages data flow, we need to understand the key components that make up its architecture. Here’s a visual overview of how data flows from Producers to Brokers (split into Topics and Partitions) until it’s pulled by Consumers:

flowchart TD
    subgraph Klien["Client Applications"]
        P1["Producer A"]
        P2["Producer B"]
        C1["Consumer Group 1 (App X)"]
        C2["Consumer Group 2 (App Y)"]
    end

    subgraph Cluster["Kafka Cluster"]
        subgraph Broker1["Broker 1 (Server 1)"]
            T1_P0[("Topic A - Partition 0 (Leader)")]
            T1_P1_R[("Topic A - Partition 1 (Replica)")]
        end
        subgraph Broker2["Broker 2 (Server 2)"]
            T1_P1[("Topic A - Partition 1 (Leader)")]
            T1_P0_R[("Topic A - Partition 0 (Replica)")]
        end
    end

    P1 -->|Send Event| T1_P0
    P2 -->|Send Event| T1_P1

    T1_P0 -. Replication .-> T1_P0_R
    T1_P1 -. Replication .-> T1_P1_R

    T1_P0 -->|"Pull Event"| C1
    T1_P1 -->|"Pull Event"| C1
    T1_P0 -->|"Pull Event"| C2
    T1_P1 -->|"Pull Event"| C2

Let’s walk through these components one by one, conceptually:

1. Event #

An event is the smallest unit of data in Kafka. It records a fact about something that has happened in the real world or inside your application. Technically, an event in Kafka is written as a combination of a key, a value, a timestamp, and optional extra metadata called headers.

• Key: Used to determine the destination partition for a message and to identify entities (e.g., User ID).
• Value: The main content of the event, usually formatted as JSON, Avro, Protobuf, or plain text.
• Timestamp: The time the event occurred.

2. Producer #

A producer is a client application whose job is to create and send events into Apache Kafka. For example, a Payment Service acts as a producer when it sends successful transaction events to Kafka.

3. Consumer #

A consumer is a client application that subscribes to and reads events from Kafka. Unlike traditional asynchronous systems where the broker pushes data to receivers, in Kafka it’s the consumer that actively pulls data from the broker whenever it has the capacity to process it.

4. Topic #

A topic is a logical folder or category used to group similar events together. For example, you could create a topic named transaksi-sukses to hold all payment transaction data, and a topic klik-halaman to track user click activity on your website.

5. Partition #

Each topic in Kafka is not stored as one single large file. Instead, a topic is split into several smaller parts called Partitions. A partition is the physical unit of parallelism in Kafka. Events within a single partition are guaranteed to be stored in order and given a unique sequence number called an Offset.

flowchart LR
    subgraph Topic1["Topic: transaksi-sukses"]
        direction TB
        subgraph P0["Partition 0"]
            P0_0["Offset 0"] --> P0_1["Offset 1"] --> P0_2["Offset 2"] --> P0_3["Offset 3"]
        end
        subgraph P1["Partition 1"]
            P1_0["Offset 0"] --> P1_1["Offset 1"] --> P1_2["Offset 2"]
        end
        subgraph P2["Partition 2"]
            P2_0["Offset 0"] --> P2_1["Offset 1"] --> P2_2["Offset 2"] --> P2_3["Offset 3"] --> P2_4["Offset 4"]
        end
    end

6. Broker #

A broker is a physical or virtual server that runs the Apache Kafka process. Brokers are responsible for receiving messages from producers, storing them on disk, and serving data read requests from consumers. The collection of brokers that are connected and working together is called a Cluster.

Apache Kafka’s Design Characteristics #

Why is Kafka able to handle millions of messages per second with very low latency? The answer lies in the brilliant architectural design decisions made by its creators. Here are the four main pillars of Kafka’s physical design:

1. Append-Only Log System (Distributed Commit Log) #

Kafka’s internal storage structure is very simple: a binary log file that only allows writing at the end of the file (append-only). You can’t modify data that has already been written (immutable), and you can’t insert data in the middle of the file.

Because it only performs sequential writes at the end of the file, the operating system can optimize it extremely efficiently. Sequential writes on a modern mechanical hard drive (HDD) can even match random-write speeds in memory (RAM), because the hard drive’s read head doesn’t need to move around searching for free sectors.

flowchart LR
    subgraph AO["Append-Only Writes"]
        direction LR
        D0["Old Data 0"] --> D1["Old Data 1"] --> D2["Old Data 2"] --> D3["New Data 3 (Only at the end)"]
        style D3 fill:#22c55e,stroke:#15803d,color:#fff
    end

2. Leveraging the Operating System Page Cache #

Many large database applications try to manage their own memory caches in user space (JVM heap). Kafka takes the opposite approach: it hands cache management entirely to the operating system through the Page Cache.

When data is written to the file system, the operating system automatically keeps it in spare memory (page cache). When a consumer reads recently written data, it’s served directly from the OS memory cache without any physical disk access. This minimizes JVM heap usage and avoids the long Java Garbage Collection pauses.

3. Zero-Copy Data Transfer #

On a traditional application server, sending a file from disk to a network socket requires a long journey:

1. Data is read from disk into the operating system’s page cache.
2. Data is copied from the page cache into the application buffer (user space).
3. Data is copied again from the application buffer into the OS socket buffer.
4. Data is copied from the socket buffer to the network interface card (NIC).

This process wastes CPU cycles because the same data is copied over and over. Kafka uses an OS function called sendfile (on Linux/Unix) that implements the Zero-Copy technique. Data is copied directly from the OS page cache to the network card without ever entering the Java application’s memory. This drastically reduces CPU usage and maximizes the network card’s bandwidth.

flowchart TD
    subgraph Traditional["Traditional Transfer Path"]
        direction LR
        Disk1["Disk"] --> PC1["Page Cache"] --> US["User Space (JVM Heap)"] --> SB1["Socket Buffer"] --> NIC1["NIC (Network Card)"]
    end
    subgraph ZeroCopy["Zero-Copy Path (Sendfile)"]
        direction LR
        Disk2["Disk"] --> PC2["Page Cache"] --> SB2["Socket Buffer"] --> NIC2["NIC (Network Card)"]
        style PC2 stroke-dasharray:5,5
        style SB2 stroke-dasharray:5,5
    end

4. Message Batching and Compression #

Instead of sending messages one by one over expensive TCP network connections, Kafka’s producer clients intelligently group several messages into a batch before sending them. Kafka brokers also store those message batches as-is without splitting them, and deliver them to consumers as one complete batch.

You can also enable data compression (such as Gzip, Snappy, Lz4, or zstd) on the producer side. Because messages are compressed in batches, the compression ratio is much higher than compressing messages individually, saving significant network bandwidth and disk storage.

Real-World Use Cases in the Industry #

With all these design advantages, where is Apache Kafka typically deployed? Here are some real use cases in today’s tech industry:

1. User Activity Tracking #

This is Kafka’s original use case at LinkedIn. Every user interaction on an app or website — button clicks, pages viewed, keyword searches, and time spent — is sent as an event to a Kafka topic. Data analytics teams then consume this data in real-time to update job recommendation algorithms or surface content that matches users’ interests at that very moment.

2. Log Aggregation & Centralized Logging #

Companies with large microservices architectures have thousands of servers generating millions of log lines every minute. Accessing logs on each server machine one by one is impossible during an incident.

You can use log collection agents (such as Filebeat or Fluentd) to read local logs on every machine and send them straight to Kafka. From Kafka, the log data flows into search and analytics systems like Elasticsearch (ELK Stack) or Splunk to make centralized monitoring easier.

3. Real-Time Data Processing (Real-Time ETL) #

In modern retail, warehouse inventory must be continuously updated. When a purchase happens at a physical store checkout or in an e-commerce app, the sales event is sent to Kafka. This data stream is then processed by a stream processing system to instantly decrement stock in the inventory system and update the sales analytics dashboards monitored by management.

4. Event Sourcing and CQRS Architecture #

In an Event Sourcing architecture, an application’s final state isn’t stored directly in a database. Instead, the application stores the entire sequence of chronological events that triggered those state changes. Kafka is ideal for this pattern because of its persistent, immutable log nature. If the main database suffers total corruption, you can rebuild the entire database state from scratch just by replaying all the events stored in Kafka from offset zero.

5. Metrics Collection and System Monitoring (Metrics & Ingest System) #

Beyond application logs, Kafka is also the backbone for collecting system performance metrics in real-time. Server metrics such as CPU usage, RAM consumption, disk I/O, network latency, and even business metrics (for example, registrations per minute) are sent to Kafka periodically. Monitoring agents (like Telegraf or Prometheus) can act as producers. This metric data is then funneled into time-series databases (like InfluxDB or Prometheus) and visualized with Grafana. With Kafka as a buffer, massive metric traffic spikes won’t overload your monitoring database, keeping performance dashboards responsive during incidents.

6. Database Synchronization via Change Data Capture (CDC) #

In modern architectures, you often need to replicate data from the primary operational relational database (OLTP) to analytical databases (OLAP or Data Warehouse) instantly without burdening the primary database. The CDC (Change Data Capture) technique handles this by listening to changes directly on the database’s transaction log (such as MySQL’s binlog or PostgreSQL’s Write-Ahead Log). CDC tools (like Debezium) capture every change (insert, update, delete) and send them to Kafka as an event stream. Consumer services then read those events to update search databases (like Elasticsearch) or load them into a Data Warehouse (like BigQuery or Snowflake) in real-time.

flowchart TD
    subgraph CQRS["Event Sourcing & CQRS with Kafka"]
        App["Application"] -->|Send Transaction Event| Kafka{"Kafka Topic (Persistent)"}
        Kafka -->|Async| SQL["Process Event to SQL"]
        Kafka -->|Async| NoSQL["Process Event to NoSQL"]
        SQL --> SQL_DB[("SQL Read DB <br/> (Query Optimized)")]
        NoSQL --> NoSQL_DB[("NoSQL Search DB <br/> (Search Optimized)")]
    end

Summary #

• Event Streaming Platform — Apache Kafka was designed from the start not just as an ordinary message queue, but as a comprehensive distributed data streaming platform (Publish/Subscribe, Store, and Process).
• Born at Massive Scale — It was born at LinkedIn in 2010 to solve the complexity of custom data pipelines and the throughput limits of traditional message brokers.
• The Key to Extreme Speed — Kafka’s remarkable speed is powered by its append-only binary log storage structure, optimal use of the OS page cache, Zero-Copy data transfer, and message batching and compression.
• Broad Use Cases — It’s ideal for real-time activity tracking, centralized log aggregation, stream data processing (Real-Time ETL), and modern architectures like Event Sourcing and CQRS.