Size-Based Retention #

Managing physical storage space availability on broker clusters is one of the most critical responsibilities for Apache Kafka system administrators. Relying only on Time-Based Retention is very risky in production. Imagine our system suddenly experiences a data traffic spike 10 times higher due to a sales promotion or DDoS attack. Within hours, a massive data volume fills broker hard disk capacity before the retention time limit (e.g., 7 days) is reached. If disk capacity hits 100%, the Kafka broker suffers a fatal failure (crash) and disrupts the entire cluster operation. To anticipate this disaster, Kafka provides a second rescue policy: Size-Based Retention.

Dissecting the Size Retention Parameters: Per-Partition Limits #

There are two main parameters controlling size-based retention, both at the global cluster configuration level and the custom topic configuration level.

1. log.retention.bytes (Partition Size Limit) #

This parameter sets the maximum disk space capacity allowed for one single partition, not the entire topic total! This is a classic trap that often confuses system engineers.

Topic Total Disk Usage Calculation Formula #

If we have a topic named user-clicks configured with:

  • partitions = 12
  • replication.factor = 3
  • log.retention.bytes = 10,737,418,240 (10 GB)

Then, the maximum total disk capacity that topic can consume across the entire cluster is:

$$\text{Total Disk} = \text{log.retention.bytes} \times \text{Partitions} \times \text{Replication Factor}$$

$$\text{Total Disk} = 10 \text{ GB} \times 12 \times 3 = 360 \text{ GB}$$

So, even though we set the number 10 GB on the parameter, we must ensure our cluster infrastructure has at least 360 GB of free space specifically for that topic.

2. log.segment.bytes (Segment Size Limit) #

This parameter determines the maximum size of one physical log segment file before it’s closed (rolled). By default, Kafka sets:

$$\text{log.segment.bytes} = 1,073,741,824 \text{ bytes (1 GB)}$$

Meaning, every time an active log file reaches 1 GB in size, it’s closed into a read-only file, and Kafka opens a new segment to continue writing.

Logical Interaction: “Whichever Comes First” #

In an ideal production environment, we’re highly recommended to enable both the time retention policy (log.retention.ms) and size retention (log.retention.bytes) simultaneously on sensitive topics.

Kafka evaluates both policies with the “Whichever Comes First” principle.

Simulation Example Scenario #

We have a topic configured with:

  • log.retention.ms = 604800000 (7 Days)
  • log.retention.bytes = 5368709120 (5 GB per partition)

Let’s review two different data flow conditions:

  • Case A: High Throughput (Traffic Spike) The incoming data rate is very heavy at 3 GB per day. On the second day, total data in the partition reaches 6 GB. Because the 5 GB size limit is passed before the 7-day limit is reached, Kafka immediately triggers deletion of the oldest inactive segments to bring the partition total size back under 5 GB. The oldest data here only survives less than 2 days before being deleted.
  • Case B: Low Throughput The data rate is very slow, only 500 MB per day. After 7 days, total collected data is only 3.5 GB. Even though the 5 GB size limit hasn’t been reached, because the message age in the oldest segment has passed the 7-day limit, Kafka immediately deletes that segment.

This dual principle provides layered protection: securing disk capacity from overload during data surges, while also guaranteeing old unused data is cleaned up on time for data governance compliance.

Size-Based Segment Deletion Evaluation Anatomy #

To understand how Kafka evaluates log file trimming, we must see how segment sizes accumulate. Let’s illustrate the physical partition directory structure on the broker disk:

flowchart TD
    subgraph Partisi_Disk["Physical Partition Folder: orders-0/"]
        direction LR
        S1["00000000.log <br> (Inactive Segment 1) <br> Size: 1 GB <br> Status: Deleted"]
        S2["00005000.log <br> (Inactive Segment 2) <br> Size: 1 GB <br> Status: Safe"]
        S3["00010000.log <br> (Inactive Segment 3) <br> Size: 1 GB <br> Status: Safe"]
        S4["00015000.log <br> (Active Segment) <br> Size: 800 MB <br> Status: Immune"]
    end

    subgraph Evaluasi["Size Retention Evaluation Logic"]
        direction TB
        Limit["log.retention.bytes Limit = 2.5 GB"]
        Sum["Total Partition Size = 3.8 GB"]
    end

    S1 -->|"Trimmed by Log Cleaner"| Trash["Free OS Disk Space"]
    style S1 stroke:#c62828,stroke-width:2px
    style S2 stroke:#2e7d32,stroke-width:2px
    style S3 stroke:#2e7d32,stroke-width:2px
    style S4 stroke:#0288d1,stroke-width:2px

Log Trimming Evaluation Steps by the Broker #

  1. Active Segment Exception: Same as time-based retention, the active segment file (00015000.log in the diagram above) is never deleted, regardless of its size.
  2. Total Size Calculation: The broker calculates the total size of all segment files (both active and inactive) in that partition folder. From the diagram:

$$\text{Total Size} = 1 \text{ GB} + 1 \text{ GB} + 1 \text{ GB} + 0.8 \text{ GB} = 3.8 \text{ GB}$$

  1. Capacity Excess Detection: The broker compares the calculation result with the log.retention.bytes parameter limit. Because the total size ($3.8 \text{ GB}$) is larger than the configuration limit ($2.5 \text{ GB}$), a limit violation occurs.
  2. Gradual Deletion: The broker starts deleting the oldest inactive segment files in order. First, the 00000000.log file (1 GB) along with its index files is removed from disk.
  3. Re-evaluation: The broker recalculates the partition total size:

$$\text{New Total Size} = 1 \text{ GB} + 1 \text{ GB} + 0.8 \text{ GB} = 2.8 \text{ GB}$$

Because the new total size ($2.8 \text{ GB}$) is still larger than the $2.5 \text{ GB}$ limit, the broker continues deleting the next oldest inactive segment on the next cycle, or immediately deletes until the total size is truly below the $2.5 \text{ GB}$ threshold.

JBOD (Just a Bunch of Disks) & Multi-Volume Storage Management #

In enterprise-scale production environments, a single Kafka broker is often configured to use several physical disks simultaneously without combining them through RAID architecture (JBOD - Just a Bunch of Disks). This is achieved by defining multiple directory paths in the server.properties file separated by commas:

log.dirs=/mnt/disk1/kafka-data,/mnt/disk2/kafka-data,/mnt/disk3/kafka-data

How Does Disk Selection Impact Retention? #

When using JBOD configuration:

  1. New Partition Placement: When a new partition is created, Kafka places it on the disk directory with the fewest partitions, not based on remaining free disk capacity!
  2. Asymmetric Disk Risk: If one partition receives a very large data throughput, that disk (e.g., /mnt/disk1/) can fill up much faster than other disks, triggering more frequent size retention evaluations on the partitions residing there.
  3. Moving Partitions: We can use the kafka-log-dirs.sh CLI tool to evaluate disk usage on each volume. Since Kafka 2.0+, we can use the alterReplicaLogDirs API (via Java programs or shell scripts) to dynamically move overloaded partitions between physical disks in the same broker without downtime and without burdening inter-broker network bandwidth.
  4. Offline Volume Metric: We must monitor the JMX OfflineLogDirectoryCount metric to detect failures in one of our JBOD disk volumes so a single disk failure doesn’t cripple the entire broker, but only disables the affected partitions.

Calculating Physical Memory Overhead: Index File Pre-Allocation #

Besides the main .log data file, every active segment is also accompanied by .index and .timeindex index files. The maximum size of these index files is dynamically set by the broker using the parameter:

log.index.size.max.bytes=10485760 # 10 MB (Default)

Index Allocation Overhead Trap #

When a new segment becomes the active segment, Kafka immediately performs a full 10 MB physical pre-allocation for the .index file and 10 MB for .timeindex on the OS disk, regardless of whether the message data inside is still empty.

  • Empty Partition Overhead: If we have 10,000 active partitions on one broker, this index pre-allocation automatically consumes free disk space of:

$$10,000 \text{ partitions} \times 20 \text{ MB (Indexes)} = 200 \text{ GB}$$

This 200 GB of disk space is immediately consumed by empty index files. We must include this variable in capacity calculations so size-based retention isn’t triggered prematurely from drastically reduced OS free disk space.

Preventing Disk Saturation Using Producer Quotas #

Applying size retention is a reactive protection step after data enters the broker. A better preventive step is limiting the ingestion rate from the producer side using the Quotas feature.

We can limit the maximum throughput producers can write (for example, 10 MB/second per client) to ensure rogue producers can’t fill cluster disks before the Log Cleaner Thread gets a chance to run its deletion tasks.

Setting Producer Quotas with CLI #

# Limiting the write rate of client-id 'payment-app' to 10 MB/s (10485760 bytes/s)
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type clients \
  --entity-name payment-app \
  --alter \
  --add-config producer_byte_rate=10485760

Linux Filesystem Optimization: Segment File Pre-Allocation #

When Kafka creates a new segment file on a Linux file system (like XFS or ext4), by default that file is allocated dynamically as data grows. However, under high throughput workloads, this can cause physical file fragmentation on the disk, degrading sequential disk I/O performance.

To solve this, Kafka provides the configuration:

log.preallocate=true

How Does Pre-Allocation Help Retention? #

When this property is enabled, during a new segment roll, the broker immediately reserves storage space equal to the log.segment.bytes maximum limit (e.g., 1 GB) wholesale on the OS disk from the start, filling it with empty bytes.

  • Benefit: The OS guarantees physical data blocks on disk are allocated sequentially, avoiding fragmentation.
  • Important Note: This feature makes the active segment file’s physical size on disk appear as 1 GB immediately from creation, even though its contents are still empty. Therefore, we must prepare mature backup disk capacity because the disk space calculation is immediately and aggressively consumed by these new empty active segments.

Filesystem Mount Option Recommendation #

To improve write performance and large segment file evaluation, ensure the XFS filesystem mount option is configured in /etc/fstab with:

# Optimizing mount options for sequential disk data writes
/dev/sdb1 /var/lib/kafka/data xfs noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k 0 2

Setting noatime and nodiratime removes the OS overhead of updating file read timestamps every time a message is pulled by consumers.

Dual Policy Collaboration: cleanup.policy=compact,delete #

Since modern Kafka versions, we can combine two log cleanup policies simultaneously on the same topic:

cleanup.policy=compact,delete

How the Combination Works #

In this combined mode:

  1. The Log Compaction process runs periodically to compact inactive log segments, leaving only the latest data status per key.
  2. At the same time, Size-Based (or Time-Based) Retention still evaluates the total partition size.
  3. If the total compacted partition size still exceeds the log.retention.bytes limit, Kafka permanently deletes the oldest compacted segments.

This dual policy is very useful for scenarios where we need key state compaction for daily data savings, but still demand absolute historical data cleanup if total data exceeds our cluster’s physical disk capacity.

Emergency Runbook: Handling Full Broker Disks (Disk Saturation) #

If an emergency condition occurs where one broker’s disk hits 95% due to delayed data cleanup or capacity miscalculation, here are the emergency mitigation steps we can safely execute.

Step 1: Identify the Largest Topics #

Find which partition folders consume the most disk space using Linux disk commands:

# Finding the 10 largest partition folders in the Kafka log path
du -h --max-depth=1 /var/lib/kafka/data | sort -hr | head -n 10

Step 2: Apply Strict Size Retention Dynamically #

Don’t try to manually delete .log files using Linux rm commands! Manually deleting log files corrupts broker indexes, crashes the Kafka Java process, and damages KRaft/ZooKeeper metadata.

The correct solution is force-triggering the Log Cleaner Thread by dynamically lowering the target topic’s size retention parameter to a very low limit (for example, 2 GB per partition):

# Aggressively lowering the retention limit to trigger instant cleanup
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name nama-topik-raksasa \
  --alter \
  --add-config log.retention.bytes=2147483648

Within at most 5 minutes (per the log.retention.check.interval.ms interval), the Log Cleaner Thread scans that topic, detects the size violation, and deletes stale segments to safely free disk space.

Summary #

  • Size-Based Retention: Protects broker physical storage media from total disk capacity exhaustion during data surges.
  • Per-Partition Limit: The log.retention.bytes property applies to one single partition, not the total accumulation of the entire topic.
  • JBOD (Multi-Disk): The multi-directory log.dirs configuration allows disk load division without RAID, but requires detailed per-volume capacity monitoring.
  • Evaluation Synergy: When time and size retention are enabled together, Kafka uses the “whichever comes first” principle.
  • Pre-Allocation: Setting log.preallocate=true helps prevent file fragmentation on Linux file systems by reserving the full segment capacity upfront.
  • Index Overhead: Every active segment pre-allocates .index and .timeindex files of 10 MB each on disk, producing disk overhead for large partition counts.
  • Quotas: Use the producer_byte_rate parameter to preventively hold back rogue producers from filling disk space.
  • Policy Combination: Setting cleanup.policy=compact,delete compacts key data while also absolutely limiting the partition’s physical storage size.

← Previous: Time-Based Next: Log Compaction →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact