Partition #

If a topic is the logical category of a data stream in Apache Kafka, then Partition is the smallest physical unit where that data is actually stored, replicated, and consumed. Partitions are the main engine behind the horizontal scalability, high parallelism, and fault tolerance that make Kafka the industry standard for large-scale data processing. Understanding the internal partition mechanism in depth will help us design system architectures capable of handling millions of messages per second without performance bottlenecks.

Basic Concept: What is a Partition? #

Physically, a partition is a single ordered append-only log file stored on the local disk of one Kafka broker server. When a producer sends an event to a partition, the new data is always appended at the very end of the log file. This characteristic makes write operations to Kafka very fast (time complexity $O(1)$) because it doesn’t require random index lookups on disk like traditional databases.

Every event written to a partition gets a unique sequence number called an Offset. Offsets are 64-bit integers that increase monotonically (0, 1, 2, …). The Offset acts as the physical address of the data within the partition. Once an event receives a particular offset, that number is absolute and will never change or be reused for another event, even if older data before that offset has been deleted by the retention policy.

The ordering guarantee in Kafka is local per partition, not global per topic. That means Kafka guarantees consumers will read messages in exactly the order they were sent only if all those messages are in the same partition. If messages are spread across different partitions, the broker can no longer guarantee read order.

Why Do We Need Partitions? (Parallelism & Scalability) #

To understand the importance of partitions, let’s imagine a topic without partitions (only 1 single partition). That topic could only be stored on one broker server’s disk. When data volume grows to hundreds of terabytes, that server runs out of disk space. Additionally, only one consumer application can read the data at a time to preserve message order, limiting our processing speed.

By splitting a topic into multiple partitions, Kafka solves these two big problems at once:

1. Horizontal Scaling #

Partitions of the same topic don’t have to be stored on the same broker server. Kafka automatically spreads those partitions across the various broker servers available in the cluster.

For example, if we have a topic with 6 partitions and a cluster with 3 brokers, each broker manages 2 partitions. When our cluster’s storage capacity starts filling up, we simply add a new broker server to the cluster and move some partitions to that new server without shutting down the system (zero-downtime).

2. Consumer Parallelism #

Partitions act as the workload division unit for consumers. In Kafka, one partition may only be consumed by at most one consumer within one Consumer Group at a time.

If we have a topic with 4 partitions, we can run up to 4 consumer applications in parallel to process that data simultaneously. If we increase the number of consumers to 5, the 5th consumer will sit idle because all partitions have already been allocated to the previous 4 consumers. Therefore, the number of partitions sets the maximum parallelism limit for our downstream data processing.

How to Determine the Ideal Number of Partitions #

Determining the partition count when creating a new topic is a critical architectural decision. If there are too few partitions, we’ll hit performance bottlenecks because we can’t add more consumers to speed up data processing. Conversely, too many partitions waste cluster memory managing partition metadata and slow down recovery time when a server fails.

To calculate the partition count scientifically and precisely, we can use the target throughput formula below:

$$P = \max\left(\frac{\text{Target Throughput}}{T_p}, \frac{\text{Target Throughput}}{T_c}\right)$$

Let’s break down each variable in the formula:

  • Target Throughput: The total data volume we want to handle per second. Example: $100\text{ MB/second}$ or $100,000\text{ messages/second}$.
  • $T_p$ (Maximum Producer Throughput): The maximum speed of one producer application thread writing data to one partition. In general, a well-optimized uncompressed Kafka producer can write about $50 - 100\text{ MB/second}$ to a single partition.
  • $T_c$ (Maximum Consumer Throughput): The maximum speed of one consumer application reading and processing data from one partition. This speed heavily depends on the business complexity inside our consumer code (for example, whether it must write to another database or call third-party REST APIs). An average consumer can process about $10 - 20\text{ MB/second}$ per partition.

Calculation Example: #

A logistics company wants to process GPS coordinate data from its courier fleet with a target throughput of $150\text{ MB/second}$. Through local testing, they found that one producer can send $50\text{ MB/second}$ per partition, while their consumer (because it must do fairly complex processing into a PostgreSQL database) can only process about $15\text{ MB/second}$ per partition.

  • Producer throughput requires: $\frac{150}{50} = 3\text{ partitions}$
  • Consumer throughput requires: $\frac{150}{15} = 10\text{ partitions}$

Taking the maximum of both results, we need at least 10 partitions for that topic to safely handle the $150\text{ MB/second}$ workload without triggering consumer lag.

Workload Distribution (Partition Routing) #

How does a producer direct event flows into the right partition efficiently? Let’s study the physical message distribution flow:

flowchart TD
    subgraph Producer["Producer Application"]
        EventData["Event: Account_ID = 1024 <br/> Payload = Rp 500.000"]
        Partitioner{"Partitioner Engine"}
    end

    subgraph KafkaCluster["Kafka Cluster (Topic: 3 Partitions)"]
        BrokerA["Broker 1"] ---> Partisi0[("Partition 0")]
        BrokerB["Broker 2"] ---> Partisi1[("Partition 1")]
        BrokerC["Broker 3"] ---> Partisi2[("Partition 2")]
    end

    EventData --> Partitioner
    Partitioner -->|"MurmurHash2(1024) % 3 <br/> = Partition 1"| Partisi1

Broadly speaking, there are two built-in mechanisms for routing events to partitions:

  1. Keyed Message: If the event has a key, the producer computes the Murmur2 hash of that key and takes the modulo with the number of active partitions. All data with the same key is guaranteed 100% to land in the same physical partition.
  2. Non-Keyed Message: If the key is null, the producer uses the Sticky Partitioning strategy. Messages are grouped into one network batch and efficiently sent to the same random partition until the batch fills up, before switching to another partition.

Key Skewness (Hot Partition) Impact and How to Avoid It #

Using keyed message delivery often creates serious problems in production if we don’t choose keys wisely. This problem is known as Key Skewness or Hot Partition.

Key skewness happens when a particular key value dominates real-world business transaction volume. For example, if we set partner_id as the partition key in an e-commerce payment application, and one of our partners (say a giant merchant like Shopee) contributes 80% of all transactions in our application, then the partition that is the hash destination for Shopee’s key will process 80% more data than other partitions.

Failure Impact: The broker storing that hot partition suffers congestion from disk I/O and CPU exhaustion. Consumers assigned to read that partition experience high consumer lag, while consumers reading other partitions sit idle.

Mitigation Solution: Key Salting #

To avoid hot partitions without losing the benefits of message ordering, we can apply the Key Salting technique. We add a random value (salt) behind the main key to spread data evenly across several different partitions:

# ANTI-PATTERN: Using a low-entropy key directly without processing
# This causes data to pile up on a single partition if one partner dominates transactions.
def kirim_transaksi_skewed(producer, partner_id, data):
    # The pure "Shopee" key will always land in the exact same partition
    producer.send('topik-pembayaran', key=partner_id.encode('utf-8'), value=data)

# The CORRECT solution: Applying Key Salting
# We add a controlled random number behind the key to split the workload across several partitions.
def kirim_transaksi_salted(producer, partner_id, data):
    import random
    
    # If a giant merchant is detected, add a random salt of 1 to 3
    if partner_id == "Shopee":
        salt = random.randint(1, 3)
        kunci_salted = f"{partner_id}_{salt}"
    else:
        kunci_salted = partner_id

    # Shopee data is now spread evenly across 3 different hash partitions
    # Downstream consumers must strip the "_salt" suffix when processing data
    producer.send('topik-pembayaran', key=kunci_salted.encode('utf-8'), value=data)

Consequences of Adding Partitions to a Running Topic #

In day-to-day operations, we might be tempted to immediately add partitions to an active topic when detecting a traffic spike. Kafka does support adding partitions dynamically (on-the-fly). However, this action has very serious architectural impacts that developers often overlook:

1. Broken Key-to-Partition Ordering Guarantee #

As we discussed, producers map keys to partitions using the hash modulo formula:

$$\text{Partition} = \text{MurmurHash2}(\text{Key}) \pmod{\text{Number of Partitions}}$$

If we change the divisor — the Number of Partitions (for example, from 3 partitions to 5 partitions) — the modulo result for the same key automatically changes.

  • Before Adding (3 Partitions): An event with key User_99 produces a hash pointing to Partition 1. All User_99 historical data is neatly stored in Partition 1.
  • After Adding (5 Partitions): New events for User_99 are sent by the producer. Since the divisor is now 5, the formula produces a new route to Partition 4.

As a result, User_99’s old data is in Partition 1, while new data is written to Partition 4. Our consumer application reading data in parallel processes the new data before finishing the old data, totally breaking the data ordering guarantee for that entity.

2. Impact on Stateful Processing (Kafka Streams / KTable Join) #

If we use stream processing libraries like Kafka Streams to aggregate data (for example, calculating running balances per user) or join two different data streams by key, adding partitions mid-stream triggers local state corruption (state store). Data that should gather on one processing thread managing a particular partition now splits to another thread, causing inaccurate calculation results.

Safe Migration Strategy: #

If we absolutely must increase a topic’s throughput capacity and can’t tolerate broken data ordering:

  • Create a New Topic: The best solution is to create a new topic with a larger partition count from the start (for example prod.finance.payment-v2).
  • Switch Traffic: Redirect producers to write to the new topic, then run new consumer applications reading from the new topic.
  • Drain Old Data: Let old consumers process all remaining data left in the old (v1) topic until it’s clean (drain), before fully deactivating them.

Replication and Leader-Follower Partitions #

To prevent data loss when a physical server fails, Kafka replicates every partition to several different broker servers. The number of replicas is set using the Replication Factor configuration parameter (for example, replication factor = 3).

In this replication mechanism, partitions are managed using the Leader-Follower role:

  • Leader Partition: Among all partition replicas, only one replica is designated as the Leader. All read and write activity from producers and consumers is by default directed to this Leader partition.
  • Follower Partition: Other replicas act as Followers. They don’t serve read/write requests from general clients; their only job is to replicate data asynchronously from the Leader partition to keep their data state in sync.
  • In-Sync Replicas (ISR): The group of replicas (including the Leader) whose data state is truly aligned and in sync with the Leader. If the Leader fails, only ISR members are eligible to be elected as the new Leader.

Common Mistakes (Anti-patterns) in Designing Partitions #

There are several partition design mistakes we must avoid to keep the Kafka cluster running stably:

1. Designing High-Load Topics with Only 1 Partition #

Developers who want global data ordering guarantees often create topics with just 1 partition for all workloads.

Consequences: This caps the topic’s throughput at the maximum I/O limit of a single broker server. Additionally, we can never increase consumer-side processing speed because adding more than one consumer in a consumer group is pointless (other consumers sit idle). Global ordering guarantees must be sacrificed for scalability by splitting topics into multiple partitions and using the right logical keys.

2. Creating Too Many Partitions Without Calculation (Over-Partitioning) #

Some developer teams immediately create hundreds or thousands of partitions for all topics assuming “more partitions means faster”.

Consequences: Every partition in Kafka is represented by a directory file structure at the broker OS level. Having too many active partitions triggers:

  • OS file descriptor consumption exceeding safe limits (open file limits).
  • Dramatically increased JVM Heap memory consumption on broker servers to manage partition metadata.
  • Very high latency during cluster failover because KRaft/ZooKeeper must re-elect thousands of partition Leaders simultaneously when one broker dies.
  • As a production rule of thumb, keep the maximum partition count around 100 partitions per broker and no more than 20,000 partitions per cluster.

Summary #

  • Partition Definition — A partition is an ordered append-only log file on a broker’s local disk, acting as the smallest physical unit of horizontal scalability, data replication, and parallelism in Apache Kafka.
  • Data Offsets — Every piece of data in a partition is uniquely identified by a sequence number called an Offset that is immutable and monotonically increasing.
  • Ordering Guarantee — Message ordering guarantees in Apache Kafka are local per partition, not global across the whole topic.
  • Consumer Parallelism — The number of partitions determines the maximum parallelism capacity of our consumer applications within one Consumer Group.
  • Ideal Partition Formula — Calculate the ideal partition count by comparing the system’s target throughput against the maximum throughput of one producer ($T_p$) and one consumer ($T_c$).
  • Hot Partition Mitigation — Avoid the Key Skewness (hot partition) phenomenon by choosing keys with high random distribution, or use Key Salting for merchants/partners with dominant transaction volumes.

← Previous: Topic Next: Producer →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact