Topic Design #

In the Apache Kafka ecosystem, a Topic is a logical category or container wrapping a message stream (event stream). Although creating a topic sounds like a very simple administrative task, topic structure and configuration design is actually an important architectural decision determining the performance, security, and scalability of our entire data pipeline. Bad topic design — like non-standard naming, unplanned partition allocation, or misconfigured data retention policies — can trigger operational chaos, memory leaks, and even loss of critical data in production.

Through this article, we’ll deeply explore industry-standard domain-based topic naming rules (domain-driven topic naming), dissect time and size based data retention policy tuning, choose the right cleanup policy between delete vs compact, and formulate a structured guide for safe production-scale topic creation.

Topic Naming Standardization (Topic Naming Conventions) #

One of the most common operational problems in large enterprise Kafka clusters is the accumulation of hundreds of topics with irregular names, like test-topic, logs, data-baru, or my_topic_123. This haphazard naming makes it difficult for infrastructure teams to monitor cluster statistics, apply security rules (ACLs), and sort out which data is still in use or already expired.

To design a clean, manageable cluster, we must apply Domain-Driven Design based naming standardization. The recommended industry-standard naming format is:

$$\text{Format: } \langle\text{domain}\rangle.\langle\text{subdomain}\rangle.\langle\text{entity}\rangle.\langle\text{event-name}\rangle$$

Let’s break down each component above:

  • domain — The highest-level business area representation (for example, payment, customer, telemetry, logistic).
  • subdomain — The sub-system or bounded context representation within that domain (for example, billing, verification, iot, shipping).
  • entity — The name of the main business object represented by the data (for example, invoice, password, sensor, package).
  • event-name — The specific event name representing the message’s historical fact, usually written in past-tense verb form to indicate an event stream (for example, created, changed, logged, delivered).

Examples of Good Topic Names: #

  1. payment.billing.invoice.created — The data stream of newly issued billing invoices in the payment sub-system.
  2. customer.identity.password.changed — Reporting user password change events for security needs.
  3. telemetry.sensor.temperature.logged — Temperature reading log data from IoT devices.

Internal System Topic Naming Rules #

One absolute rule we must obey is never create application topics with a double underscore (__) prefix. Topic names starting with two underscores are reserved specifically for Apache Kafka’s internal system needs.

Examples of Kafka’s built-in internal topics include:

  • __consumer_offsets — Stores consumer offset commit history.
  • __transaction_state — Stores client asynchronous transaction state.
  • _schemas — Used by the Confluent Schema Registry to manage data schemas.

Additional Naming Rules: #

  • Use Lowercase: Always use lowercase to avoid typing ambiguity. Kafka is case-sensitive, so topics Payment and payment are considered two different topics.
  • Use Hyphens: If an entity or event name consists of two words, use a hyphen - to separate them (for example, invoice-created). Avoid using the dot character . as a word separator because dots are dedicated as logical domain level delimiters.
  • Don’t Include Environment Names: Avoid including words like dev, staging, or prod in topic names (for example, prod.payment.billing.invoice-created). Environments should be separated at the physical cluster or logical cluster namespace level, not inside the topic name itself. Including environments in topic names forces us to change application code configurations when migrating across environments.

Data Retention Policy: Time vs Size #

Unlike traditional message queue systems that immediately delete messages once read by consumers, Apache Kafka retains messages on broker local disk. This message lifespan is controlled through data retention configuration.

We can set data retention based on two main parameters:

1. Time-Based Retention (retention.ms) #

This parameter determines how long a message is stored in the broker’s log segments before becoming eligible for deletion. The default value for this parameter is 604800000 milliseconds (7 days).

  • Tuning for Sensitive Data: For financial transaction data or audit logs requiring long history traceability (event sourcing), we can raise this value to 30 days, 1 year, or even unlimited (-1).
  • Tuning for High-Volume Data: For IoT telemetry data or application performance metrics whose value expires quickly, we’re advised to lower this value to 1 day (86400000 ms) or just a few hours to save broker disk capacity.

2. Size-Based Retention (retention.bytes) #

This parameter determines the maximum storage capacity in bytes for one partition (not the total topic size overall!). The default value is -1 (unlimited).

  • If we set this parameter to 1073741824 bytes (1 GB) on a topic with 3 partitions, the topic’s maximum log capacity on the broker is 3 GB.
  • When the log size on one partition exceeds 1 GB, Kafka immediately deletes the oldest log segment file, even if the retention.ms retention time for messages in that segment hasn’t expired.
  • Disk Capacity Protection: Setting this parameter is mandatory in production as a safety net so broker disks don’t fill up (disk space protection) from unexpected message volume spikes.

The Often-Confusing Segment Rolling Trap #

Many developers get confused when data in their topic isn’t deleted even though the retention.ms retention time has passed (for example, set to 1 day, but data from 3 days ago still exists). Why does this happen?

Kafka only performs data cleanup on log segments that are already closed (closed segment). Kafka never deletes data from log segments still actively accepting writes (active segment).

Two parameters controlling segment closure are:

  • log.segment.bytes (default 1 GB) — Segments close when they reach 1 GB in size.
  • log.roll.hours (default 7 days / 168 hours) — Segments close when they reach 7 days old.

If our data volume is very small, the active segment file may need 7 days to reach 1 GB in size or hit the 7-day time limit. During those 7 days, the segment stays actively open, and messages inside it are never deleted by the system, even if we set retention.ms to 1 day.

To overcome this, if we set very short retention times, we must balance it by tuning the log.roll.hours parameter to a smaller value (for example 2 hours) so segments close and get cleaned quickly.

Cleanup Policy: Delete vs Compact #

When designing topics, we must choose how brokers treat historical data that has passed retention limits through the cleanup.policy configuration:

1. cleanup.policy = delete (Default) #

This is the standard cleanup policy. When log segments pass the time or size retention limits, the entire oldest physical segment files are deleted directly from the broker disk.

  • Use Case: Best suited for daily transaction data streams, clickstream activity tracking logs, infrastructure metrics, and other temporal events with limited validity periods.

2. cleanup.policy = compact (Log Compaction) #

This policy enables the Log Compaction feature. When active, Kafka guarantees that for every unique message Key, the broker always retains at least one latest message (latest state) in the log. Old messages with the same key are cleaned by the Log Cleaner background thread.

This compaction process runs in the background based on the min.cleanable.dirty.ratio configuration (default 0.5 or 50%). This means the cleanup process only runs after the number of new (dirty) messages in the log reaches at least 50% of the total compacted log size.

  • Deletion with Tombstones: If we want to permanently delete a key on a compacted topic, we must send a message with that key and set its Value to null. This marker message is called a Tombstone Message. Kafka propagates this tombstone to all consumers, then removes the key from disk on the next cleanup cycle.
  • Use Case: Ideal for storing latest-state snapshot data, like user profile tables (storing the latest address by User ID), product inventory status (storing latest stock by product SKU), or customer account balances.

Visualizing Topic, Partition, and Broker Structure #

For a concrete understanding of how a domain-driven topic splits into physical partitions and gets redundantly distributed across cluster brokers, look at the visual diagram below:

flowchart TD
    subgraph LogicalView["Logical View (Client Applications)"]
        Topic["Topic: payment.billing.invoice.created <br/> (Replication Factor = 2)"]
    end

    subgraph PhysicalView["Physical View (Broker Cluster)"]
        direction LR
        subgraph Broker1["Broker Server 1"]
            P0_L["Partition 0 <br/> (Leader)"]
            P1_F["Partition 1 <br/> (Follower)"]
        end
        subgraph Broker2["Broker Server 2"]
            P1_L["Partition 1 <br/> (Leader)"]
            P0_F["Partition 0 <br/> (Follower)"]
        end
    end

    Topic -->|Partition 0| P0_L
    Topic -->|Partition 1| P1_L

    P0_L -. Replication .-> P0_F
    P1_L -. Replication .-> P1_F

    style Topic fill:#ddffdd,stroke:#88ff88
    style P0_L fill:#ddffdd,stroke:#88ff88
    style P1_L fill:#ddffdd,stroke:#88ff88

Common Mistakes (Anti-patterns) in Topic Design #

Here are some fatal error patterns in topic creation and management in the industry, along with their fixes:

1. Enabling Automatic Topic Creation (auto.create.topics.enable = true) #

The broker default configuration auto.create.topics.enable is true. This allows brokers to instantly create new topics when a client writes to or reads from an unregistered topic name.

Consequences: If our application developers misspell a topic name in code (for example, writing payment.billing.invoice.creaated instead of payment.billing.invoice.created), Kafka automatically creates a new topic with broker default configurations (usually just 1 partition and replication factor 1).

  • New messages enter that typo topic unnoticed.
  • Official consumers listening to the real topic never receive that data.
  • The typo topic has no failure tolerance (replication = 1), so if the broker dies, the data is lost forever.
  • Disable this auto-creation in production by setting auto.create.topics.enable = false, and manage topic creation declaratively through automation scripts.

2. Code Scenario: Creating Topics Safely Through the Admin Client #

Instead of relying on auto-creation or manually creating topics through typo-prone command-line terminals, we’re advised to create topics using the Java AdminClient API as part of our application deployment (CI/CD) automation flow.

Here’s a comparison of the wrong and right topic creation implementations:

// =========================================================================
// ANTI-PATTERN: Relying on auto.create.topics.enable in Production
// Clients directly send data to topics without formal creation validation.
// =========================================================================
public void sendToUncreatedTopic(KafkaProducer<String, String> producer, String payload) {
    // ✗ DON'T: Send directly to a random topic name. If auto-create is active,
    // the broker creates a topic with minimum default specs (Very Dangerous!).
    ProducerRecord<String, String> record = new ProducerRecord<>("my-dirty-logs-topic", payload);
    producer.send(record);
}

// =========================================================================
// THE CORRECT SOLUTION: Formal Topic Initialization Using AdminClient
// Topics are declared programmatically with safe capacity parameters.
// =========================================================================
import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.common.config.TopicConfig;

import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class TopicProvisioner {
    private final AdminClient adminClient;

    public TopicProvisioner(String bootstrapServers) {
        Properties props = new Properties();
        props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        this.adminClient = AdminClient.create(props);
    }

    public void createStandardTopic(String topicName, int numPartitions, short replicationFactor) {
        // ✓ CORRECT: Set retention configuration explicitly for disk safety
        Map<String, String> topicConfigs = new HashMap<>();
        topicConfigs.put(TopicConfig.RETENTION_MS_CONFIG, "2592000000"); // Time retention: 30 Days
        topicConfigs.put(TopicConfig.RETENTION_BYTES_CONFIG, "10737418240"); // Size retention: 10 GB per partition
        topicConfigs.put(TopicConfig.CLEANUP_POLICY_CONFIG, TopicConfig.CLEANUP_POLICY_DELETE);

        NewTopic newTopic = new NewTopic(topicName, numPartitions, replicationFactor)
                .configs(topicConfigs);

        try {
            // Run topic creation synchronously
            adminClient.createTopics(Collections.singletonList(newTopic)).all().get();
            System.out.printf("✓ Successfully created standardized topic: %s (Partitions: %d, Replicas: %d)%n", 
                topicName, numPartitions, replicationFactor);
        } catch (InterruptedException | ExecutionException e) {
            if (e.getCause() instanceof org.apache.kafka.common.errors.TopicExistsException) {
                System.out.println("Topic already registered, skipping creation step.");
            } else {
                System.err.println("Failed to create topic: " + e.getMessage());
            }
        }
    }

    public void close() {
        adminClient.close();
    }
}

By applying the topic provisioning code above, our cluster is guaranteed to only run topics with mature partition capacity configurations and safe replication factors for failure tolerance.

Summary #

  • Domain-Driven Naming — Apply industry-standard business domain based topic naming: [domain].[subdomain].[entity].[event-name] for administrative order.
  • retention.ms — The parameter controlling how long messages stay on the broker (default 7 days), must be adjusted based on business data sensitivity characteristics.
  • retention.bytes — The maximum memory limit per partition on the broker disk (default -1). Acts as a safety net so server disks don’t fill up from message spikes.
  • cleanup.policy=delete — Deletes all binary log segment files that have passed retention limits to free disk space quickly.
  • cleanup.policy=compact — Enables Log Compaction to retain the latest status message for each message key, discarding old historical change records.
  • auto.create.topics.enable=false — Mandatory broker configuration in production to prevent automatic default topic creation from client-side topic name typos.
  • AdminClient API — Use a programmatic API to declare new topic creation automatically and consistently as part of the code deployment pipeline.
Next: Partition Strategy →
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact