Log Compaction #

When we talk about data retention in Apache Kafka, our minds often immediately jump to absolute data deletion based on time or file size limits. However, there are many business scenarios where we don’t want to discard historical data at all, but only want to keep the final state of an entity. Imagine a bank account balance system, user profile updates, or a product stock database. We don’t need the entire year-old change history consuming terabytes of storage; we only need to know the customer’s current latest balance or the user’s active email address. To solve this architectural need, Kafka provides an amazing feature called Log Compaction.

Basic Philosophy: Delete Policy vs Compact Policy #

Architecturally, Kafka divides the commit log cleanup policy (cleanup.policy) into two main types:

  1. cleanup.policy=delete (Default) Kafka physically deletes all old log segment files as soon as their age exceeds the time limit or their size exceeds the byte limit. Messages are deleted without regard to keys or contents.
  2. cleanup.policy=compact Kafka guarantees that for every message key, the broker always retains at least one last message with the largest offset. Old messages with the same key are gradually cleaned up to save disk space.

By using the compaction policy, we can store data permanently (infinite retention) per business key without worrying about disk storage filling up with stale duplicate data.

How Does the Log Cleaner Work? (Clean vs Dirty Portion) #

The log compaction process isn’t run in real-time as messages are written by producers; it runs asynchronously in the background by a special thread called the Log Cleaner Thread.

Log Segment Structure Division #

To understand the compaction process, we must see how a partition’s log file is divided into two logical areas:

flowchart LR
    subgraph Clean["Clean Portion (Compacted)"]
        direction LR
        C1["Key A: Off 10"] --- C2["Key B: Off 12"] --- C3["Key C: Off 15"]
    end
    
    subgraph Dirty["Dirty Portion"]
        direction LR
        D1["Key A: Off 20"] --- D2["Key D: Off 21"]
    end
    
    Clean <--> |"First Dirty Offset"| Dirty
  • Clean Portion: The initial log area already compacted in previous cycles. This area only contains one unique message per key.
  • Dirty Portion: The later log area containing new messages just arrived from producers. In this area, the same key can appear multiple times with different values.
  • First Dirty Offset: The boundary point marking the end of clean data and the start of dirty data.

Compaction Cycle Stages #

When the Log Cleaner Thread detects that the dirty data ratio has exceeded the minimum threshold set by the log.cleaner.min.cleanable.ratio parameter (default: 0.5 or 50% dirty data):

  1. Building a Skimpy Offset Map: The Log Cleaner scans all inactive segments in the Dirty Portion area. It builds a hash table in broker memory called a Skimpy Offset Map. This table maps a 128-bit hash of the message key to its largest offset found in the dirty area.
  2. Selectively Copying Messages: The Log Cleaner starts re-reading the log file from the beginning (from the Clean Portion area to the end of the Dirty Portion).
    • For every message record read, the broker matches its key with the data in the Skimpy Offset Map.
    • If the offset of the message being read is smaller than the offset recorded in the map for that key, the message is considered stale and immediately ignored (discarded).
    • If the message offset equals the largest offset in the map, the message is written to a new clean segment file.
  3. Segment Merge: After copying finishes, the old dirty and clean segments are physically merged into a new dense segment. The old segments are deleted from the OS disk, and the First Dirty Offset pointer is moved to the end of the newly formed clean segment.

Memory Analysis: Skimpy Offset Map & Performance Tuning #

Because the Skimpy Offset Map is built directly in the broker’s JVM Heap memory, we must calculate memory capacity carefully to avoid causing the broker to run out of memory (OutOfMemoryError).

Per-Entry Memory Structure #

Each entry in the Skimpy Offset Map consumes 24 bytes of memory divided into:

  • 16 bytes: For storing the MD5 hash of the message key.
  • 8 bytes: For storing the message offset number (64-bit long data type).

Real Case Calculation (RAM Sizing) #

If we have a dirty partition with 10,000,000 unique keys (10 million unique keys), the minimum RAM required to process that offset map is:

$$\text{Memory} = 10,000,000 \times 24 \text{ bytes} = 240,000,000 \text{ bytes} \approx 240 \text{ MB}$$

Log Cleaner Tuning Parameters #

To optimize the log cleaner’s performance, we can tune the following parameters in the server.properties file:

  • log.cleaner.dedupe.buffer.size: The total memory size allocated to hold the Skimpy Offset Map across all cleaner threads (default: 134,217,728 / 128 MB). For clusters with millions of unique keys, we must raise this parameter (e.g., to 1,073,741,824 / 1 GB).
  • log.cleaner.threads: The number of background threads handling log compaction (default: 1). If we have many compacted partitions, increase this thread count (e.g., 4 or 8) to speed up compaction cycles.
  • log.cleaner.io.buffer.size: The disk I/O buffer size used to read/write segment files during compaction (default: 2,097,152 / 2 MB).

Avoiding I/O Spikes with Bandwidth Throttling #

The compaction process constantly reads and rewrites large segment files on disk, which can clog the disk I/O bus. This can disrupt producers writing new messages. To limit the Log Cleaner’s write bandwidth, we can set the parameter:

# Limiting the Log Cleaner read/write speed to a maximum of 15 MB/second per broker
log.cleaner.io.max.bytes.per.second=15728640

This setting ensures the compaction process runs stably in the background without ever seizing the entire disk I/O throughput from clients’ main transactional operations.

Log Compaction Limits & Consequences for Consumer Applications #

Applying the log compaction policy brings several important architectural implications that our consumer applications must anticipate.

1. Non-Contiguous Offsets #

After compaction runs, message offsets in the log file are no longer linearly sequential (for example: from 1, 2, 3, 4 changing to 1, 4, 12, 15 because offsets 2 and 3 were deleted).

  • Consumer Behavior: Consumers can still read data normally. When a consumer seeks to offset 2, Kafka intelligently directs the consumer’s read pointer to the next available offset, which is 4.

2. Null Message Keys Prohibition #

Log Compaction only works on messages that have a key.

  • Null Key Danger: If we send a message with a null key to a compacted topic, the Kafka broker fails to map that message into the Skimpy Offset Map. This can cause errors in the Log Cleaner Thread or cause the message to skip the compaction process, permanently clogging storage space.

Visual Log Compaction Flow #

Here’s a visual diagram showing the commit log state before and after the compaction process runs by the Log Cleaner Thread:

flowchart TD
    subgraph Sebelum_Pemadatan["1. Log Before Compaction (Many Key Duplicates)"]
        direction LR
        K1_O1["Key: User_A <br> Offset: 10 <br> Val: Jakarta"]
        K2_O2["Key: User_B <br> Offset: 11 <br> Val: Bandung"]
        K1_O3["Key: User_A <br> Offset: 12 <br> Val: Surabaya"]
        K3_O4["Key: User_C <br> Offset: 13 <br> Val: Medan"]
        K2_O5["Key: User_B <br> Offset: 14 <br> Val: Bali"]
    end

    subgraph Sesudah_Pemadatan["2. Log After Compaction (Only Largest Offsets Remain)"]
        direction LR
        K1_O3_A["Key: User_A <br> Offset: 12 <br> Val: Surabaya"]
        K3_O4_A["Key: User_C <br> Offset: 13 <br> Val: Medan"]
        K2_O5_A["Key: User_B <br> Offset: 14 <br> Val: Bali"]
    end

    K1_O1 -.->|"Deleted (Offset 12 exists)"| Sesudah_Pemadatan
    K2_O2 -.->|"Deleted (Offset 14 exists)"| Sesudah_Pemadatan
    K1_O3 -->|"Retained"| K1_O3_A
    K3_O4 -->|"Retained"| K3_O4_A
    K2_O5 -->|"Retained"| K2_O5_A

    style K1_O1 stroke:#c62828,stroke-width:2px
    style K2_O2 stroke:#c62828,stroke-width:2px
    style K1_O3 stroke:#2e7d32,stroke-width:2px
    style K3_O4 stroke:#2e7d32,stroke-width:2px
    style K2_O5 stroke:#2e7d32,stroke-width:2px

Key Deletion Mechanism: Tombstone Marker #

If the compaction system always guarantees at least one last message per key is retained forever, then how do we permanently delete a key from a compacted topic? For example, when a user deletes their account, we must delete all their personal data to comply with data privacy regulations (like GDPR).

To solve this, Kafka introduces the Tombstone Marker concept (also known as a Delete Marker).

How Does a Tombstone Work? #

  1. Sending a Null Value: The producer client sends a new message with the key we want to delete, but sets the message value to null.
  2. Recorded as a Tombstone: When the broker receives this null-valued message, it’s written to the log and officially considered a Tombstone Marker.
  3. First-Stage Cleanup: When the next log compaction cycle runs, the Log Cleaner Thread detects the tombstone marker. The broker deletes all old messages related to that key, leaving only the tombstone marker file itself in the log.
  4. Final-Stage Cleanup: The tombstone marker must not be deleted immediately, because active consumers reading data offline/lagging need to see the tombstone to know that key has been deleted.
  5. Retention Duration Parameter: The tombstone marker is stored on disk for the duration configured by the parameter:

$$\text{log.cleaner.delete.retention.ms} = 86,400,000 \text{ ms (24 Hours)}$$

After 24 hours pass, on the next compaction cycle, the tombstone marker file is totally deleted from disk. The key is now truly gone from the Kafka cluster.

Main Use Cases: CDC & Kafka Streams State Stores #

Log Compaction is the architectural foundation behind modern distributed data modules.

1. Change Data Capture (CDC) #

In microservices architectures, we often use CDC tools (like Debezium) to capture table changes in transactional databases (like PostgreSQL or MySQL) and replicate them to Kafka.

  • CDC topics must be configured with cleanup.policy=compact. We only need to record the latest database row state per primary key. If the database updates a row, new data with the largest offset in Kafka represents the current database row, while old data can be safely discarded to save disk space.

2. Changelog Topics in Kafka Streams (Recovery Time Estimation) #

Real-time data processing applications (like Kafka Streams or Apache Flink) often store their business state in local in-memory databases (like RocksDB) called State Stores.

  • To guarantee fault tolerance, the state store periodically sends its change log to a hidden topic in Kafka called a Changelog Topic, which is compacted.
  • If the server running the Streams application suddenly dies, a replacement server can recover its local memory state very quickly by replaying data from the beginning of that compacted changelog topic.

Recovery Speed Increase Calculation #

Let’s calculate the real recovery time difference for a Streams application processing 1,000,000 daily transactions (total uncompacted raw data of 10 GB):

  • Scenario A: Without Log Compaction (Delete Policy) The application must replay all 10 GB of change logs from the beginning. With an average client network I/O speed of 16 MB/second, the time needed to rebuild the local state store is:

$$\text{Recovery Time} = \frac{10,240 \text{ MB}}{16 \text{ MB/s}} \approx 640 \text{ seconds (10.6 Minutes)}$$

  • Scenario B: With Log Compaction Because the topic is compacted, the 10 GB of data containing repeated updates shrinks to only the latest state per customer (for example, only 100 MB of clean data remains). With the same I/O speed:

$$\text{Recovery Time} = \frac{100 \text{ MB}}{16 \text{ MB/s}} \approx 6.2 \text{ seconds}$$

Final Result: Log Compaction cuts our microservice application recovery downtime by 99% (from 10 minutes to only 6 seconds), which is crucial for maintaining operational Service Level Agreements (SLAs).

Operational CLI Guide #

Here are practical command-line commands for configuring and managing log compaction.

1. Creating a New Compacted Topic #

# Creating a user profile topic with the log compaction policy
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create \
  --topic user-profiles \
  --partitions 3 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config log.cleaner.min.cleanable.ratio=0.3 \
  --config min.insync.replicas=2

2. Sending Tombstone Messages via Console Producer #

To permanently delete the user profile with key user_id_99, we must send a null-valued message. In the console producer, we can use a special property flag to define null:

# Enabling the console producer with key parsing and custom null marker
kafka-console-producer.sh --bootstrap-server localhost:9092 \
  --topic user-profiles \
  --property parse.key=true \
  --property key.separator=: \
  --property null.marker=TOMBS

# Type the following command in the terminal (Key: user_id_99, Value: null)
user_id_99:TOMBS

3. Configuring Minimum Compaction Lag #

By default, Kafka compacts dirty data as soon as the ratio limit is exceeded. However, if our consumer applications need time to read the entire real-time update history (not just the final state), we can postpone compaction of new messages for a certain period using log.cleaner.min.compaction.lag.ms:

# Holding messages from being compacted for at least 1 hour (3,600,000 ms)
# so real-time consumers have time to read the full historical change sequence.
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name user-profiles \
  --alter \
  --add-config log.cleaner.min.compaction.lag.ms=3600000

Summary #

  • Log Compaction: The log cleanup policy retaining at least one latest data entry with the largest offset for every message key.
  • Dirty Ratio: Compaction is triggered asynchronously when the dirty portion data ratio exceeds the log.cleaner.min.cleanable.ratio parameter (default: 0.5).
  • Skimpy Offset Map: The broker memory data structure consuming 24 bytes per entry to record the largest key offset. Adjust log.cleaner.dedupe.buffer.size as needed.
  • Cleaner Throttling: Use the log.cleaner.io.max.bytes.per.second parameter to limit disk I/O usage so it doesn’t disrupt producer transactions.
  • Tombstone Marker: A null-valued message sent by producers to trigger permanent key deletion on compacted topics.
  • Offset Seeks: Post-compaction, log offsets are no longer linearly contiguous, but the consumer search API still works normally by skipping empty offsets.
  • Use Case: Log Compaction is a vital foundation for CDC database synchronization systems and state store changelogs in Kafka Streams.
← Previous: Size-Based
About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact