Topic #

In the Apache Kafka ecosystem, a topic is the most fundamental concept, acting as a logical container or category for grouping similar event streams. If we analogize Kafka to a traditional relational database, a topic is like a table. If we analogize it to a file system, a topic acts like a storage folder. Understanding how topics are designed logically, managed physically at the broker level, and configured optimally is the key to preventing future failures in our data architecture.

Basic Concept: What is a Topic? #

Logically, a topic in Apache Kafka acts as a channel or data pipeline where producers send events and consumers read that data. Topics are multi-producer and multi-consumer. That means a topic can receive data streams from many different producer applications simultaneously, and at the same time, the data inside it can be read independently by many consumer groups with different business purposes.

The most unique characteristic of topics in Kafka is their non-destructive read behavior. In traditional message broker systems like JMS (Java Message Service) or RabbitMQ, when an application reads a message from a queue, the message is removed from the queue so it can’t be read by other applications.

In Kafka, events written to a topic are permanent and aren’t deleted right after being read. Data stays in the topic until the retention period you set is exceeded. This allows one same data stream (for example, a sales transaction stream) to be consumed by the Finance Service for bookkeeping, the Inventory Service to update stock, and the Analytics Service for real-time business dashboards without interfering with each other.

Internal Topic Structure: Logical vs Physical #

It’s important to understand the difference between the logical representation of a topic and its physical storage implementation inside the Kafka cluster. Logically, we see a topic as a single complete linear data stream from start to finish. However, physically behind the scenes, Kafka splits the topic into several smaller parts called Partitions.

Let’s visualize how a single topic is divided into several physical partitions distributed across various broker servers in the Kafka cluster:

flowchart TD
    subgraph Logis["Logical View (Client Applications)"]
        TopicLogis["Topic: transaksi-pembayaran <br/> (Unified Event Stream)"]
    end

    subgraph Fisik["Physical View (Kafka Cluster)"]
        subgraph Broker1["Broker 1 (Server A)"]
            Partisi0[("Partition 0 <br/> (Log File 0)")]
        end
        subgraph Broker2["Broker 2 (Server B)"]
            Partisi1[("Partition 1 <br/> (Log File 1)")]
        end
        subgraph Broker3["Broker 3 (Server C)"]
            Partisi2[("Partition 2 <br/> (Log File 2)")]
        end
    end

    TopicLogis --> Partisi0
    TopicLogis --> Partisi1
    TopicLogis --> Partisi2

The physical division into partitions is the main secret behind Apache Kafka’s outstanding throughput performance and nearly unlimited horizontal scalability. Each partition is an ordered append-only log file stored on the local disk of one Kafka broker server.

By splitting a topic into several partitions, Kafka can distribute data storage and processing load across all broker servers in the cluster. We’re no longer limited by the disk storage capacity or I/O capability of a single server.

Topic-Level Data Retention Policy #

Unlike traditional databases that store all data permanently by default, Kafka is designed as a temporal data stream processing system. We manage the data lifecycle inside topics through retention policies. This policy can be configured independently for each topic in our cluster.

There are two main methods Kafka uses to determine when data in a topic should be cleaned up:

1. Time-Based Retention #

This policy is set through the retention.ms configuration parameter. This parameter determines how long an event may stay in a partition before being deleted. The cluster default is usually 168 hours (7 days).

For example, if we set retention.ms to 86400000 (24 hours), an event that arrives on Monday at 08:00 will be automatically deleted from the broker’s disk on Tuesday at 08:00. This policy is perfect for recurring streaming data whose usefulness declines over time, such as application metric logs or courier fleet GPS coordinate data.

2. Size-Based Retention #

This policy is set using the retention.bytes parameter. This parameter limits the maximum accumulated data size of a partition on disk. It’s important to note that this limit applies per partition, not per entire topic.

For example, if a topic has 3 partitions and we set retention.bytes to 10737418240 (10 GB), the total storage limit for the topic is 30 GB across the cluster. When the log segment file size in one partition exceeds 10 GB, Kafka deletes the oldest data segments in that partition until it’s back under the safe limit. This policy is crucial for preventing our servers from running out of disk space due to unexpected data spikes.

Cleanup Policy: Delete vs Compact #

Kafka provides the cleanup.policy parameter to determine what should happen when the retention limit is reached:

• delete (Default): Kafka permanently deletes all old data that has passed the time or size retention limit from disk.
• compact (Log Compaction): Instead of raw deletion, Kafka scans the partition and keeps only one event with the latest value (latest state) for each unique Key. Intermediate historical data is discarded, but the last known state of an entity is never lost. This policy is ideal for event-driven architectures like storing user profiles or financial account balances.

Topic Naming Convention #

In medium to large organizations adopting microservices architectures, the number of Kafka topics in a production cluster can grow very quickly to hundreds or even thousands. Without a strict Topic Naming Convention agreed upon from the start, our cluster will soon become a confusing data jungle that’s hard to manage and vulnerable to security leaks.

A good naming convention must be descriptive, consistent, easy for new developers to understand, and designed to support wildcard-based security configuration (like ACL authorization).

The widely recommended topic naming standardization format in the industry is as follows:

$$\langle\text{environment}\rangle.\langle\text{domain}\rangle.\langle\text{subdomain}\rangle.\langle\text{data_type}\rangle.\langle\text{event_name}\rangle$$

Let’s break down each component that makes up a topic name:

• Environment: Indicates the cluster where the topic lives. Examples: prod (production), staging (testing), or dev (development).
• Domain: The name of the department or main bounded context of our business. Examples: finance, logistic, or marketing.
• Sub-domain: The specific sub-system under the main domain. Examples: payment, billing, or tracking.
• Data Type: Describes the kind of data content in the topic. Common values are:
  • fact: Contains indisputable records of past events (facts), usually append-only (immutable event log).
  • command: Contains asynchronous instructions or requests to do something.
  • cdc: Records of direct changes from the main database (Change Data Capture).
• Event Name: A specific description in past-tense verb form explaining the event. Examples: transaction-completed, user-registered, or order-dispatched.

Topic Naming Comparison Examples: #

• BAD (Too generic, no context):
  • transaksi
  • data-user
  • kafka-test
• GOOD (Structured, descriptive, and safe):
  • prod.finance.payment.fact.transaction-completed
  • prod.logistic.tracking.cdc.courier-location
  • staging.marketing.campaign.command.send-newsletter

With a structured format like the one above, Kafka Administrator teams (SRE) can very easily apply ACL (Access Control List) security authorization rules automatically. For example, they can write a single rule: “The Finance Service is allowed to read all topics with the prod.finance.* prefix pattern”.

Critical Topic-Level Configurations #

Although Kafka brokers have global default configurations for all topics, there are many use cases where we need to customize specific per-topic parameters to fit particular performance needs or data security levels.

The table below summarizes the most important topic-level configuration parameters we must know and manage wisely:

Common Mistakes (Anti-patterns) in Topic Management #

When designing and managing topics in a production Apache Kafka cluster, there are several common anti-patterns that developers often perform and that can be fatal to the stability and performance of our cluster.

1. Wildly Enabling Topic Auto-Creation #

By default, Kafka brokers are configured with auto.create.topics.enable = true. That means if a producer application sends a message to a topic name that was never actually created in the cluster, the broker automatically creates that topic using default parameters (usually just 1 partition and replication factor = 1).

Consequences: Developers can accidentally misspell topic names in their code (for example, sending to prod.finance.paymment with two ’m’s). Kafka will automatically create that misspelled topic. This litters the cluster’s topic list, degrades performance, and most dangerously, sensitive data gets sent to a topic without safe replication configuration and isn’t monitored by any consumer.

# ANTI-PATTERN: Relying on dynamic topic auto-creation in production
# Misspelling a topic name will trigger the creation of a new, unsafely configured topic.
def kirim_event_auto_create(producer, event_data):
    # Misspelling the topic name 'payment' as 'paymeent'
    # If auto-create is active, the broker will create a rogue topic with minimal, unsafe specs
    producer.send('prod.finance.paymeent', value=event_data)

# The CORRECT solution:
# 1. Disable auto-creation in the broker's server.properties: auto.create.topics.enable = false
# 2. Always create topics declaratively using an admin tool (CLI or Terraform) before running the app
# 3. Handle send failure exceptions in the application code if the destination topic isn't found
def kirim_event_aman(producer, event_data):
    try:
        producer.send('prod.finance.payment', value=event_data)
    except Exception as e:
        # Handling the error if the topic isn't registered in the cluster
        logger.error(f"Failed to send data! Destination topic not registered in Kafka: {e}")
        simpan_ke_antrean_darurat(event_data)

2. Creating Too Many Dynamic Topics (Topic Per Entity) #

Some developer teams accustomed to NoSQL databases try to apply the same design pattern to Kafka by dynamically creating one topic per unique entity (for example, creating topics user.topic.ID_USER_123, user.topic.ID_USER_456, etc.).

Consequences: Every topic split into several partitions requires file handle allocation on the broker OS, metadata coordination in KRaft/ZooKeeper, and memory buffers on the client application side. Creating thousands to millions of dynamic topics will crash our Kafka cluster from memory exhaustion (Out Of Memory) and trigger severe metadata coordination congestion. Topics must always be static based on business categories, not dynamic based on user IDs.

Summary #

• Topic Definition — A topic is a logical category wrapping similar event data streams, multi-producer and multi-consumer, with non-destructive read behavior (data isn’t deleted right after being consumed).
• Logical vs Physical — A topic is a logical abstraction on the application side. Physically inside the broker server cluster, a topic is split into several Partitions as ordered append-only log files on local disk.
• Retention Policy — The data lifecycle inside a topic is managed based on a time limit (retention.ms) or a per-partition storage size limit (retention.bytes).
• Cleanup Policy — You can set the log segment cleanup policy via cleanup.policy with the delete option (fully delete old data) or compact (keep only the last state of each unique key).
• Naming Standardization — Always apply a structured topic naming convention (example: <env>.<domain>.<subdomain>.<type>.<event_name>) to ease governance and cluster ACL security configuration.
• Disable Auto-Create — Make sure the auto.create.topics.enable parameter is set to false in production clusters to prevent rogue topics created by producer application code typos.

← Previous: Event Next: Partition →