Kafka on Bare Metal: Optimizing Physical Infrastructure Performance #

When we plan Apache Kafka architectures for petabyte-scale data with millions of messages per second throughput, choosing the base infrastructure is a crucial step. Although the cloud computing and virtualization era offers provisioning convenience, running Apache Kafka directly on physical servers without intermediate layers (Bare Metal) remains the gold standard for high-performance, latency-sensitive clusters.

Running Kafka on Bare Metal eliminates all the hypervisor tax overheads commonly found on virtual machines (VMs). Without hardware resource competition (resource contention), Kafka brokers can directly manipulate disk platters, page cache memory allocations, and network packet queues with consistent latency. However, this peak performance doesn’t come by itself. We must be able to design hardware specifications precisely, tune file systems, arrange JBOD configurations, and correctly isolate NUMA memory access.

In this guide, we’ll dissect Bare Metal vs Virtualization performance comparisons, arrange minimum production cluster hardware specifications, guide optimal XFS file system configurations, explore why JBOD is superior to RAID for Kafka, and do BIOS and Linux kernel configurations for NUMA optimization.

Why Choose Bare Metal for Kafka? #

To understand why Bare Metal is superior, we must see how hypervisors on virtual machines affect Kafka’s internal workings. Kafka is an I/O-intensive system heavily depending on linear disk read-write performance and page cache memory manipulation by operating systems.

Here are three main virtualization challenges we can solve by switching to Bare Metal:

1. CPU Steal Time and Latency Jitter #

In shared virtual environments, physical CPUs are divided among several VMs. If neighbor VMs experience load spikes, our Kafka broker VM can experience CPU Steal Time—a condition where VMs want to execute CPU instructions but must queue because hypervisors are scheduling tasks for other VMs. These micro queue pauses trigger latency jitter that can damage KRaft/ZooKeeper watchdog heartbeats and disrupt ISR stability.

2. Disk Queue Contention #

On hypervisors, disk read-write requests from guest OSes must pass through virtual drivers before hypervisors translate them to physical disk controllers. If several VMs randomly write to the same storage systems, Kafka’s fast linear write performance degrades from dense I/O queues at hypervisor levels (IOPS bottlenecks).

3. Page Cache Efficiency Degradation #

Kafka uses the sendfile() system call (Zero-Copy technique) to send data from log files directly to network sockets through OS page caches. On virtual machines, data must pass through additional memory abstraction layers managed by hypervisors. These repeated virtual memory address translation processes reduce data transfer efficiency and increase CPU loads.

Production Hardware Specification Recommendations #

When choosing physical hardware for Kafka broker servers, we must avoid unbalanced specification allocations. For example, renting 64-core CPUs but only supplying 16 GB RAM is wasteful, because performance immediately stalls from too-small cache memory.

Here’s a balanced hardware specification guide for every production-scale Kafka broker:

1. Processors (CPU) #

• Recommendation: 16 to 32 physical Cores (e.g., AMD EPYC or Intel Xeon Gold series).
• Characteristics: Prioritize processors with high per-core clock speeds and integrated AES-NI encryption instruction support for handling SSL/TLS transit encryption without burdening I/O processing threads.

2. Memory (RAM) #

• Recommendation: 64 GB to 256 GB ECC RAM.
• Characteristics: Allocate only 4 GB to 6 GB for Kafka broker JVM heaps. The rest (more than $90%$ of RAM) should be left free for Linux operating system page caches to hold the hottest log segments in memory.

3. Storage (Disk Storage) #

• Recommendation: Multi-disk SSDs (SATA/SAS) or NVMe (1 TB - 4 TB sizes per disk) installed independently (JBOD).
• Characteristics: Avoid single large-scale HDDs if clusters serve many slow consumers frequently reading historical data from disks. SSDs offer stable random IOPS preventing read congestion.

Kafka Bare Metal Server Physical Architecture #

Here’s a mapping diagram of how Bare Metal server physical components directly interact with operating systems and Kafka JVM processes without hypervisor intermediaries:

flowchart TD
    subgraph Hardware["Bare Metal Physical Server"]
        CPU["CPU: Multi-Core (Performance Mode)"]
        NUMA1["NUMA Zone 1 (Local RAM)"]
        NUMA2["NUMA Zone 2 (Local RAM)"]
        NIC["Network Card: Dual 10 Gbps (Bonding)"]
        DiskBus["PCIe / SATA Bus Controller"]
        
        subgraph JBOD["JBOD Disk Configuration"]
            Disk1["Disk 1: NVMe SSD (/mnt/data1)"]
            Disk2["Disk 2: NVMe SSD (/mnt/data2)"]
            Disk3["Disk 3: NVMe SSD (/mnt/data3)"]
        end
    end

    subgraph OS["Linux Operating System (XFS Filesystem)"]
        PageCache["OS Page Cache (Free RAM)"]
        NetStack["Linux TCP Network Stack"]
    end

    subgraph JVM["Java Virtual Machine"]
        KafkaBroker["Kafka Broker Process (6GB Heap)"]
    end

    CPU --> JVM
    NUMA1 --> PageCache
    NUMA2 --> PageCache
    JVM -->|"Zero-Copy (sendfile)"| PageCache
    PageCache --> NetStack
    NetStack --> NIC
    DiskBus --> JBOD
    PageCache -->|"Asynchronous Flush"| DiskBus

XFS File System Configuration #

Choosing and configuring filesystems on Linux operating systems has big impacts on how fast Kafka can write log segments to physical disks. It’s highly recommended to use the XFS file system instead of Ext4 for all Kafka data disks.

XFS Advantages for Apache Kafka: #

1. Dynamic Inode Allocation: XFS dynamically allocates inodes across entire file systems, unlike Ext4 which statically allocates inodes at format time. Because Kafka often creates and deletes large segment files, XFS is far more efficient at minimizing disk fragmentation.
2. Online Defragmentation: XFS supports online file defragmentation processes while file systems are actively mounted, maintaining consistent linear write latency.
3. High-Speed Parallel Performance: XFS was designed from the start to handle multi-threaded I/O in parallel to one file system, perfectly matching Kafka architectures using multiple I/O threads (num.io.threads).

Mount Parameter Tuning (/etc/fstab) #

To maximize XFS disk write throughput, we must disable access time logging and optimize internal XFS buffers when mounting disk partitions.

Edit our /etc/fstab file and add the following mount parameters for every Kafka data disk:

# Example XFS mount configuration for the first JBOD data disk
UUID=xxxx-xxxx-xxxx    /mnt/kafka-data1    xfs    noatime,nodiratime,nobarrier,logbufs=8,logbsize=256k,largeio,inode64    0  2

Mount Parameter Analysis: #

• noatime: Disables access time logging every time files are read. Without this option, every time consumers read log segments, operating systems must write new access time metadata to disks, triggering unnecessary write operations.
• nodiratime: Similar to noatime, this option disables access time logging for directories.
• nobarrier: Disables write barriers. This option allows disks to write data to internal caches first without waiting for physical disk platter write confirmations to finish. IMPORTANT: Only enable this option if our Bare Metal servers are equipped with RAID controller cards or HBAs having backup batteries (BBU - Battery Backed Unit) or Flash-backed caches to prevent data loss during sudden power outages.
• logbufs=8 & logbsize=256k: Increase XFS transaction log buffer counts and sizes in memory before disk writes, smoothing parallel write operations.
• largeio: Instructs file systems to report larger optimal I/O sizes to operating systems, excellent for Kafka-typical large writes.

JBOD (Just a Bunch of Disks) vs RAID Disk Tuning #

When managing multi-disks on Bare Metal servers, traditional approaches usually combine all physical disks into one logical volume using hardware RAID (like RAID 5, RAID 6, or RAID 10). However, for Apache Kafka, the most recommended design is JBOD (Just a Bunch of Disks).

Why Reject RAID for Kafka Storage? #

• RAID 5 & 6 (Parity Overhead): These RAID types calculate parity data for every write operation. Every Kafka record write triggers “Write Penalty” where controllers must read old data, calculate new parity, and rewrite data. This process destroys Kafka write throughput performance.
• RAID 10 (Overkill Cost): RAID 10 offers good performance because it does mirroring and striping. However, because Kafka already does application-level data replication across broker servers (e.g., with Replication Factor = 3), using RAID 10 at local server levels means duplicating the same data 6 times! This is an extraordinary storage budget waste.

Why Choose JBOD? #

With JBOD, every physical disk is independently formatted and mounted to different directories (e.g., /mnt/kafka-data1, /mnt/kafka-data2, etc.). We then register all those directories to the log.dirs property in Kafka’s server.properties file:

# Registering JBOD multi-path directories on brokers
log.dirs=/mnt/kafka-data1,/mnt/kafka-data2,/mnt/kafka-data3,/mnt/kafka-data4

JBOD advantages in Kafka include:

1. Automatic Write Load Spreading: Kafka distributes new topic partition replicas evenly to directories having the fewest partition counts, fairly dividing disk I/O loads across all physical disks in parallel.
2. Failed Disk Resilience (Offline Log Directories): If one physical disk on JBOD experiences sector failures (bad sectors or total deaths), Kafka doesn’t shut down entire brokers. Brokers stay alive and only deactivate those corrupted log directories (with offline log directory status). Topic partitions on other healthy disks keep serving clients normally, while partition replicas on corrupted disks are automatically diverted to other brokers by controllers.
3. Pure IOPS Maximization: We get raw I/O speed accumulations from all disks without bottleneck limitations from one single RAID controller.

Resource Isolation and NUMA (Non-Uniform Memory Access) Tuning #

Modern large-scale Bare Metal servers generally use multi-socket CPU architectures (e.g., one motherboard containing two physical processors). This architecture uses NUMA (Non-Uniform Memory Access) memory designs.

In NUMA designs, RAM memory is divided into several zones (NUMA Nodes), where each zone is physically connected to one CPU socket.

flowchart TD
    subgraph Node0["NUMA NODE 0"]
        CPU0["CPU Socket 0"] <--> |"Fast Local Access"| RAM0["Local RAM Node 0"]
    end
    
    subgraph Node1["NUMA NODE 1"]
        CPU1["CPU Socket 1"] <--> |"Fast Local Access"| RAM1["Local RAM Node 1"]
    end
    
    CPU0 <--> |"Fast Bus"| CPU1
    RAM0 <--> |"Slow Access"| RAM1

If Kafka JVM threads running on CPU Socket 0 try accessing page cache data in Local RAM Node 1, they must pass through slow processor inter-connector buses (QPI/UPI interconnects). This cross-node memory access latency is far slower than local memory access, triggering program execution latency increases.

NUMA Optimization Steps for Kafka: #

1. Set BIOS Node Interleaving to Disabled #

Make sure the Node Interleaving (or NUMA mode) feature is enabled at server BIOS levels (set to Disabled or active NUMA mode, not UMA/SPAN mode). This ensures operating systems can clearly see and divide memory zones.

2. Configure Zone Reclaim Mode in Linux Kernels #

By default, if a NUMA zone runs out of free memory, Linux aggressively tries seizing local memory pages (page reclaiming) rather than allocating memory from other NUMA zones. This reclaiming process triggers disruptive micro I/O pauses.

Disable zone reclaim mode by adding the following parameter to the /etc/sysctl.conf file:

vm.zone_reclaim_mode = 0

Setting this value to 0 orders operating systems to allow cross-zone NUMA memory allocations if local memory is full, instead of aggressively forcing local cache cleanups.

3. Run Kafka with Interleave Memory Policies #

To prevent one NUMA zone from experiencing memory scarcity from large Kafka page cache monopolies, we’re advised to spread Kafka JVM memory allocations evenly across all NUMA zones using the numactl utility.

Edit our Kafka service systemd initialization script (/etc/systemd/system/kafka.service):

[Service]
Type=simple
User=kafka
Group=kafka
# Run Kafka with the numactl --interleave=all instruction
ExecStart=/usr/bin/numactl --interleave=all /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
LimitNOFILE=100000

The --interleave=all policy forces operating systems to allocate JVM memory pages alternately (round-robin) across all active NUMA zones, evenly dividing memory loads and eliminating crash risks from allocation scarcity on one specific local memory zone.

Bare Metal Hardware Readiness Audit Checklist #

Do the following verification steps on our physical servers before starting Apache Kafka software installations to make sure Bare Metal compliance standards are met:

Summary #

• Free from Virtualization Overheads — Running Kafka on Bare Metal eliminates latency jitter from CPU steal times and disk I/O queue contention common on VM hypervisors.
• XFS File System Tuning — Use XFS as Kafka data file systems by including noatime and nodiratime mount flags to eliminate access time metadata write overheads.
• Choose JBOD over RAID — Leverage JBOD to economically increase pure disk IOPS and enable Kafka’s automatic offline directory failure handling features.
• Apply NUMA Interleave — Run broker JVM processes using the numactl --interleave=all command to evenly divide memory loads across all physical NUMA zones and prevent localized memory crashes.