Anti-Patterns in Production: 10 Critical Mistakes and a Readiness Audit Checklist #

Running Apache Kafka in large-scale production environments demands high precision on operating system configurations, hardware, Java Virtual Machine (JVM), and the broker’s own internal parameters. Many organizations assume Kafka can run at maximum performance just using out-of-the-box defaults. Under high real-world workloads, this assumption often collapses instantly and triggers costly business production outages.

Based on experience managing Kafka clusters at various industry scales, we can identify continuously recurring configuration error patterns — which we call Anti-Patterns. In this closing article of the entire Apache Kafka tutorial series, we’ll review 10 critical anti-patterns we must avoid, pair them with concrete solutions, and arrange a comprehensive Production Readiness Checklist to make sure our systems run robustly facing workload surges.

1. JVM Heaps Set Too Large (>32 GB) #

A common misconception among developers is assuming that because Kafka brokers are written using the Java and Scala languages, we must allocate as much physical RAM as possible for JVM heaps for optimal performance.

Why Is This Dangerous? #

If we set JVM heaps too large (e.g., $32 \text{ GB}$ or $64 \text{ GB}$), Java Garbage Collectors (GCs) must manage very large memory piles. When GCs do major garbage collection cleanups, they trigger Stop-the-World (STW) processes where all broker activities stop for several seconds or even minutes. These long pauses make brokers fail to respond to heartbeats to cluster controllers, so brokers are considered dead by other quorum members, triggering broker failover turbulence and unstable partition leadership transfers (leader rebalances).

Additionally, Kafka is designed to rely on operating system file systems through OS Page Caches for log read and write operations. Spending physical RAM on JVM Heaps means robbing crucial OS Page Cache space for efficiently buffering log segment files.

# ANTI-PATTERN: Allocating almost all RAM for JVM Heaps
# export KAFKA_HEAP_OPTS="-Xmx32G -Xms32G"

# CORRECT: Limit Heaps for Kafka brokers and leave the rest of the memory for OS Page Caches
export KAFKA_HEAP_OPTS="-Xmx6G -Xms6G"

Solution: #

Limit JVM Heap allocations between $4 \text{ GB}$ to $6 \text{ GB}$ only for standard production workloads. The rest of physical RAM on servers (e.g., $58 \text{ GB}$ out of a total $64 \text{ GB}$) must be fully handed over to operating systems to be managed as Page Caches. This guarantees message reads by consumer applications can be directly served from memory caches without touching physical disks (zero-copy transmission via kernel-space).

2. Ignoring Controlled Shutdown #

Directly turning off broker servers through crude commands like kill -9 or turning off VMs without smooth transfer mechanisms is a very damaging operational anti-pattern.

Why Is This Dangerous? #

When brokers are crudely shut down, disk log index files aren’t correctly closed and partition leadership (leader partitions) statuses instantly disconnect. The absence of clean-shutdown marker files forces brokers to do slow log recovery processes (unclean recovery) at the next booting, where brokers must read and rebuild all index files (.index and .timeindex) from disk log segments. During these recovery processes, brokers can’t serve client queries, significantly increasing our system RTOs.

# ANTI-PATTERN: Letting brokers suddenly shut down without coordination
# controlled.shutdown.enable = false

# CORRECT: Instructing brokers to move leaders before shutdowns
controlled.shutdown.enable = true
controlled.shutdown.max.retries = 3
controlled.shutdown.retry.backoff.ms = 5000

Solution: #

Must enable the controlled.shutdown.enable=true parameter in every broker’s server.properties file. With this setting, when receiving shutdown signals (SIGTERM), brokers proactively move partition leadership they hold to healthy follower brokers first, then safely close all disk handler files before ending OS processes.

3. Not Limiting Min.Insync.Replicas when Acks=All #

Many teams have set acks=all (or acks=-1) on producers to ensure high data integrity, but ignore complementary configurations on broker sides.

Why Is This Dangerous? #

By default, the min.insync.replicas parameter on topics or brokers is valued at 1. If one broker dies and the cluster loses replicas leaving only 1 replica (i.e., the leader itself), producers with acks=all still receive success statuses (acknowledgments) because the minimum 1 in-sync replica requirement has been fulfilled.

If that single leader broker then experiences total hardware damage right after responding to producers, the last sent messages are lost forever from the face of the earth because they didn’t get replicated to other brokers.

# ANTI-PATTERN: Setting acks=all but leaving the default minimum replica (1)
# min.insync.replicas = 1

# CORRECT: Make sure data is stored on at least 2 in-sync replicas before success confirmations
min.insync.replicas = 2

Solution: #

To guarantee high durability levels on important business-valued topics (like financial transactions), apply the following combination rules:

• Set topic replication.factor to a minimum of 3.
• Set broker or topic min.insync.replicas to a minimum of 2.
• Use acks=all in client producer applications.

This combination guarantees every new message must be permanently written on at least 2 brokers before producer applications get success confirmations.

4. Leaving Auto-Create Topics Active #

Kafka’s built-in default configuration allows clients (producers/consumers) to automatically create topics on brokers if those topics don’t exist during initial connections.

Why Is This Dangerous? #

In production environments, leaving this option active (auto.create.topics.enable=true) damages our cluster data governance structures. Application developers wrongly writing topic names in their code (e.g., the order-topicc typo) accidentally create new unmanaged topics. These automatically created topics use broker default settings, usually with 1 partition and 1 replication factor. This leads to lost failure tolerance and performance bottlenecks because important topics run without data replication.

# ANTI-PATTERN: Letting topics be randomly created by client applications
# auto.create.topics.enable = true

# CORRECT: Block automatic topic creation and require structured manual declarations
auto.create.topics.enable = false

Solution: #

Turn off this feature by setting auto.create.topics.enable=false. Topic creation must be done explicitly and controlled through GitOps, CI/CD pipelines, or administrative kafka-topics.sh CLI commands specifying partition and replication counts agreed upon by platform teams.

5. Using RAID 5 or RAID 6 for Broker Storage #

To secure disk data from hardware damage, many infrastructure teams choose to use RAID parity configurations like RAID 5 or RAID 6 on broker servers.

Why Is This Dangerous? #

Kafka is designed to do sequential write log operations with extreme throughput. RAID 5 and RAID 6 have very large parity computation overheads when writing data (write penalties). Every time Kafka appends data to disks, RAID systems must read old data, recalculate parity values, and write new data plus parity to several physical disks. This causes write I/O latency to spike sharply, fills disk interface queues, and makes brokers slow at responding to producers.

flowchart TD
    subgraph Raid["RAID 5 / 6 (ANTI-PATTERN)"]
        direction TB
        D1["Disk 1"] & D2["Disk 2"] & D3["Disk 3"] --> |"Parity Write Overhead"| P["High Latency, I/O Jitter"]
    end
    
    subgraph Jbod["JBOD (RECOMMENDED)"]
        direction TB
        JD1["Disk 1"] --> Path1["/data1/"]
        JD2["Disk 2"] --> Path2["/data2/"]
        JD3["Disk 3"] --> Path3["/data3/"]
        Path1 & Path2 & Path3 --> R["Independent, No parity penalty"]
    end

Solution: #

Use JBOD (Just a Bunch of Disks) configurations on Kafka brokers. Kafka has built-in features for managing multi-directory storage via the log.dirs=/data1,/data2,/data3 parameter.

If one physical disk in JBOD arrangements dies, Kafka intelligently only deactivates that disk’s directory, while other disk directories keep serving normal queries. Inter-broker partition replication at the Kafka application level is enough to guarantee fault tolerance without paying performance penalties from RAID parity overhead. If absolute physical redundancy and performance are needed at hardware levels, use RAID 10.

6. Ignoring OS File Descriptor Limits #

By default, Linux distributions limit the file descriptor count one user process can open to 1024 files.

Why Is This Dangerous? #

In Apache Kafka, everything is represented as files. Every broker serves thousands of TCP socket connections from producers and consumers, and opens log segment files for every topic partition (in .log, .index, and .timeindex file forms).

If our cluster is inhabited by hundreds of topics with thousands of total partitions, this 1024 file descriptor default limit is exceeded within minutes after brokers are activated, triggering fatal java.io.IOException: Too many open files errors that suddenly stop broker processes.

# ANTI-PATTERN: Leaving narrow Linux OS default limit settings (1024)
# ulimit -n 1024

# CORRECT: Raise limits to at least 100,000 file descriptors on operating systems
# Add the following lines in the /etc/security/limits.conf file:
# kafka   soft   nofile   100000
# kafka   hard   nofile   100000

Solution: #

Edit the /etc/security/limits.conf file on broker servers to raise soft and hard file descriptor limits for users running Kafka processes (e.g., the kafka user) to at least 100000 or higher.

7. Mixing Broker & Controller Roles on Large-Scale Clusters #

In modern KRaft (Kafka Raft) architectures, brokers can act as pure Brokers, pure Controllers, or run both roles simultaneously (shared roles).

Why Is This Dangerous? #

Mixing broker and controller roles (process.roles=broker,controller) on the same physical server is very dangerous for large-scale production clusters. Busy brokers serving dense network I/O throughput and log cleanups consume CPU and RAM intensively.

If workload surges happen, controller threads on those servers run out of CPU resource cycles. As a result, controllers are slow at responding to cluster status changes, fail to detect dead brokers on time, and trigger metadata quorum instability.

# ANTI-PATTERN: Mixing data broker and metadata controller tasks on one node
# process.roles = broker, controller

# CORRECT: Separate roles to secure metadata quorum stability
process.roles = broker

Solution: #

In large-scale production clusters, physically separate roles. Provide at least 3 small dedicated servers to act as pure Controllers (process.roles=controller). Meanwhile, broker servers handling client data are set as pure Brokers (process.roles=broker). This separation guarantees cluster metadata consensus stability won’t be disturbed by client data I/O workload fluctuations.

8. Putting Operational Logs & Data Logs on the Same Physical Disk #

By default, if we don’t do directory separation configurations, application operational logs (like logs from Log4j/SLF4J libraries) are written to the same disk as Kafka topic partition data logs.

Why Is This Dangerous? #

Operational logs from Log4j (like debug audits, daily log rotations, and error logging) are synchronously written. If broker servers experience network connectivity problems, Log4j floods disks with millions of error log lines per second.

Writing operational logs to the same physical disk as partition data directories triggers disk write contention competition, lowering Kafka data log write throughput, and risking suddenly exhausting server disk capacities so brokers totally die.

# ANTI-PATTERN: Putting data logs and operational logs on the same physical disk partition
# log.dirs = /var/log/kafka/data

# CORRECT: Separate operational disk mount partitions from data log directories
# Example of separate disk mount partitions:
log.dirs = /mnt/kafka-data1/data,/mnt/kafka-data2/data
# And direct the operational LOG_DIR to the main OS disk:
# export KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=/var/log/kafka"

Solution: #

Make sure physical disks used for partition data directories (log.dirs) are dedicated disks (e.g., mounted to /mnt/kafka-data1/). Meanwhile, direct application operational log file storage directories (/var/log/kafka/) to main operating system disk partitions (OS disks).

9. Ignoring Log Cleaner Compaction Memory Sizing #

Topics using compression-based cleanup policies (log.cleanup.policy=compact) rely on broker Log Cleaner modules to remove old messages with the same keys.

Why Is This Dangerous? #

Log Cleaner modules process compaction by mapping all unique keys on log segments into memory using hash data structures. The memory size for these hashes is determined by the log.cleaner.dedupe.buffer.size parameter.

If we leave this setting at very small default values (usually $15 \text{ MB}$), brokers won’t have enough memory for mapping unique keys on large partitions. As a result, log compaction cleanup processes stall or run very slowly, making log segment files pile up uncleaned until broker disks fill.

# ANTI-PATTERN: Leaving log cleaner buffer allocations too narrow
# log.cleaner.dedupe.buffer.size = 15728640

# CORRECT: Raise log cleaner memory allocations to at least 128MB or larger
log.cleaner.dedupe.buffer.size = 134217728

Solution: #

Increase log cleanup buffer allocations by setting log.cleaner.dedupe.buffer.size=134217728 ($128 \text{ MB}$) or larger on brokers, adjusted to the unique key volumes we have in production clusters.

10. Replicating Important Topics with RF=1 #

Setting the replication.factor=1 parameter on important production topics is one of the most fatal architectural-level mistakes.

Why Is This Dangerous? #

A 1 replication factor means topic partition data is only stored on one single physical broker server. If that broker server experiences hardware failures, disk damage, or scheduled restarts, those topics immediately go offline.

Producer applications can’t send data, consumer applications can’t read data, and data stored on those corrupted disks risks permanent loss if disks can’t be recovered.

# ANTI-PATTERN: Creating production topics without backup replication
# default.replication.factor = 1

# CORRECT: Mandate at least 3 replicas for all important production topics
default.replication.factor = 3

Solution: #

Apply the absolute policy that all production topics serving business data flows must have a replication.factor of at least 3. This guarantees our clusters can lose up to 2 broker servers simultaneously without stopping those topics’ data availability.

Production Readiness Audit Checklist #

Use the audit checklist table below as the final verification gate before launching Apache Kafka clusters to production environments:

Summary #

• Allocate OS Page Caches — Don’t make JVM Heaps too large. Limit heaps in the 4-6 GB range and give the rest of physical server RAM to OS Page Caches to speed up log segment file I/O.
• Mandate Controlled Shutdown — Avoid crudely shutting down brokers. Use controlled shutdowns so leader statuses are safely transferred before brokers go offline, minimizing booting recovery times.
• Apply JBOD Without RAID Parity — Avoid RAID 5 or 6 on broker storage. Use JBOD for maximum sequential log write performance without parity computation penalties.
• Separate Controller Roles — On large clusters, separate KRaft Controller roles on dedicated physical servers to maintain metadata consensus stability during data workload surges.
• Lock Topics via GitOps — Turn off automatic topic creation and limit access rights by rejecting topic creation outside our namespace standardization rules.