Kafka on Virtual Machines: An Optimization Guide for Virtual Environments #

Running Apache Kafka directly on physical servers (Bare Metal) indeed delivers unmatched peak performance. However, in the modern computing era, many companies require infrastructure standardization using Virtual Machines (VMs) both in local data centers (using VMware vSphere or OpenStack KVM) and in Cloud Providers (like AWS, GCP, and Azure). The reasons are clear: migration flexibility, backup automation convenience, and node management efficiency.

Even though hypervisors add additional abstraction layers naturally reducing I/O efficiency, we can still run Apache Kafka reliably and at high performance inside VMs. The key is careful configuration to eliminate dynamic resource sharing (resource overcommit), minimize virtual queue latency, and tune hypervisor hardware interactions. If we treat Kafka VMs like ordinary shared-resource web server VMs, our Kafka clusters will experience fatal performance failures.

In this guide, we’ll discuss strategies for eliminating CPU Steal Time, implementing full memory reservation policies (Thick Provisioning), optimizing virtual storage architectures (PVSCSI and RDM), maximizing virtual network throughput, and choosing the right Cloud Instance types for production workloads.

Main Challenges of Running Kafka in Virtual Environments #

Before entering optimization steps, we must understand why Kafka performance often drops sharply when first deployed in virtual machines without special configurations.

Here are three main virtualization challenges we must anticipate and solve:

1. CPU Overcommit and Steal Time Impacts #

In ordinary virtualization systems, one physical CPU core can be divided into several VMs with overcommit ratios (e.g., 1:4). For Kafka, this is a disaster. If neighbor VMs use CPUs intensively, our Kafka broker VM experiences CPU Steal Time—a condition where guest operating systems want to execute CPU instructions but hypervisors delay their scheduling. As a result, data processing latency spikes and brokers can randomly disconnect from clusters for being late sending heartbeat signals (heartbeat timeouts).

2. Virtual Storage Controller Overhead #

File systems inside VMs (Guest OSes) send I/O commands to virtual disk drivers, which then forward them to virtual SCSI controllers, then hypervisors translate them to physical storage controllers (SAN/NAS/Local SSD). This chain of translation layers increases read-write latency. If virtual drivers aren’t optimized, sequential I/O performance crucial for Kafka gets hindered by virtual interface queues.

3. Virtual Network Latency Pauses (Network Jitter) #

Hypervisors share physical network cards (NICs) to several VMs using vSwitches (Virtual Switches). Software data packet switching processes inside vSwitches consume hypervisor CPU cycles and introduce micro time pauses (network jitter) absent on Bare Metal.

RAM & CPU Allocation Best Practices: Full Reservation Policies #

The first and most important rule of running Kafka in VMs is: Never share CPU and RAM resources with other VMs. We must treat Kafka as a first-class application entitled to monopolize its own physical resources.

flowchart TD
    subgraph Rec["RECOMMENDATION (Thick / Reserved)"]
        direction TB
        Cores["32 Physical Cores"] == "Reserved 100%" ==> VM1["Broker VM"]
        RAM["128 GB Physical RAM"] == "Reserved 100%" ==> VM1
        Note1["(Stable performance, zero CPU steal time, zero memory ballooning)"]
    end
    
    subgraph Anti["ANTI-PATTERN (Thin / Overcommitted)"]
        direction TB
        CoresShared["32 Shared Cores"] --> |"Shared (1:4 Ratio)"| VM2["Broker VM"]
        RAMShared["128 GB Shared RAM"] --> |"Dynamic Allocation (Ballooning)"| VM2
        Note2["(High jitter, sudden crash risks from memory contention)"]
    end

1. Absolute CPU Reservation (Zero vCPU Overcommit) #

• 1:1 Ratio: Make sure physical CPU core to virtual machine vCPU allocation ratios are set 1-to-1. If physical servers have 32 CPU cores, we may only allocate a maximum of 32 vCPUs to VMs running on those servers.
• CPU Reservation: In VMware vSphere, set the CPU Reservation parameter to maximum values (full MHz). This ensures hypervisors exclusively lock those physical CPU cores for our Kafka VM, completely eliminating CPU steal time.

2. Full Memory Reservation (Preventing Memory Ballooning) #

• Memory Reservation: Set RAM memory allocations to 100% Reserved. This forces hypervisors to physically reserve RAM when VMs first start.
• Disable Memory Ballooning & Overcommit: Dynamic RAM allocation policies like Memory Ballooning (where hypervisors seize free RAM from VMs if other VMs lack memory) must be completely disabled. If hypervisors take over free RAM from Kafka VMs, that RAM is reduced from Kafka’s OS Page Cache portion, triggering Cache Hit ratio decreases and forcing direct data reads to slow physical disks.

Hypervisor Host Kernel Optimization for Kafka #

For those of us managing local hypervisors (like KVM or ESXi), there are two mandatory hypervisor host-level optimizations for maintaining Kafka VM memory performance:

1. Disable KSM (Kernel Samepage Merging) on KVM Hosts #

KSM is a Linux kernel feature on hypervisor hosts tasked with finding identical memory pages among several VMs to then be merged into one shared memory page to save physical RAM usage.

• KSM Dangers: KSM scanning processes periodically consume very large host CPU cycles and trigger unpredictable memory access latency on Kafka VMs.
• Action: Turn off the KSM daemon on KVM hypervisor hosts by running the following commands:

  echo 0 > /sys/kernel/mm/ksm/run
  echo 0 > /sys/kernel/mm/ksm/max_page_sharing
  

2. Disable Transparent Huge Pages (THP) inside VM Guest OSes #

Even though Huge Pages sound good for database applications, the Transparent Huge Pages (THP) feature on Linux kernels inside Guest OSes often triggers dynamic memory allocation chaos for Java Virtual Machines (JVMs).

• THP Dangers: THP tries dynamically consolidating 4KB memory pages into 2MB blocks in the background. This consolidation process often triggers memory lock contention causing Kafka JVM threads to briefly stop (I/O pauses).
• Action: Turn off THP inside Kafka VMs by adding the following parameter to the grub bootloader configuration /etc/default/grub on the GRUB_CMDLINE_LINUX_DEFAULT line:

  transparent_hugepage=never
  

  Then run update-grub and restart our VMs.

VM Storage Tuning: PVSCSI and Raw Device Mapping #

The virtual storage layer often becomes the biggest bottleneck for Kafka if not configured using high-performance drivers.

1. Using PVSCSI (Paravirtual SCSI Controller) #

If we run Kafka in VMware environments, we’re advised to switch from default LSI Logic controllers to PVSCSI (Paravirtual SCSI Controller).

PVSCSI is a paravirtualized storage controller specially designed to minimize hypervisor overhead on high-throughput I/O workloads. PVSCSI advantages include:

• Reduced CPU Usage: PVSCSI processes disk I/O interrupts in large batches, significantly lowering hypervisor CPU usage for I/O translation tasks.
• Deeper Queues: PVSCSI has far deeper queue depth capacities than standard controllers.
• Queue Depth Tuning: Inside Linux Guest OSes, we can optimize PVSCSI queue depth parameters by creating the /etc/modprobe.d/vmw_pvscsi.conf file and inserting the following line:

  options vmw_pvscsi cmd_per_lun=254 ring_pages=32
  

2. Raw Device Mapping (RDM) vs VMDK #

• VMDK/VHDX (Virtual Disk Files): Provides snapshot and cloning convenience. However, these virtual disk files experience additional overhead because they’re written on top of hypervisor file systems (like VMFS or virtual NTFS).
• RDM (Raw Device Mapping): RDM bypasses hypervisor file systems and directly connects physical storage LUNs (Logical Unit Numbers) (from SAN/Local SSDs) to Guest OSes inside VMs. For consistent Kafka disk write performance equivalent to Bare Metal, we’re advised to use RDM in Physical Compatibility mode.

3. Turn Off Virtual Disk Buffering (Bypass Hypervisor Caches) #

Make sure Write Caching features at hypervisor levels for virtual disks are turned off, and install virtual disks as Independent Persistent (for VMDK-type disks). This policy ensures every data write flushed by Kafka OS page caches is directly forwarded to physical storage hardware without queuing in hypervisor internal caches.

Virtual Networks: Maximizing vNIC Performance #

Massive Kafka outbound throughput requires us to tune virtual network cards (vNICs) so they don’t stall at virtual switch levels.

1. Use VMXNET3 or VirtIO Network Drivers #

Don’t use standard network card emulation drivers like E1000. Those drivers emulate old hardware through software and are very CPU-hungry.

• In VMware environments: Always use VMXNET3 type vNICs.
• In KVM/OpenStack environments: Always use VirtIO drivers. These paravirtualized drivers integrate directly with hypervisors, support throughput up to $10 \text{ Gbps}$ or more, and have minimal CPU overhead.

2. Enable SR-IOV (Single Root I/O Virtualization) #

For giant-scale clusters, we’re advised to enable SR-IOV at physical server BIOS and hypervisor levels. SR-IOV allows physical network cards (NICs) to divide themselves into several independent virtual function indexes that can be directly installed into VMs. With SR-IOV:

• Kafka VMs communicate directly with physical network cards without passing through hypervisor vSwitches.
• Network latency pauses are drastically reduced and throughput increases equivalent to Bare Metal.

Kafka Virtual Machine Specifications #

Here’s a comparison diagram between optimally isolated Kafka VM structures vs typical resource-sharing virtual machines (anti-patterns):

flowchart TD
    subgraph Optimal["OPTIMAL KAFKA VM (Recommendation)"]
        direction TB
        subgraph ReservedResource["Locked Resources (Reserved)"]
            vCPU_Res["vCPU: 1:1 Ratio (No Overcommit)"]
            RAM_Res["RAM: 128 GB (100% Reserved)"]
        end
        subgraph VertStorage["High-Performance Storage"]
            PVSCSI["Controller: PVSCSI / VirtIO"]
            DiskRDM["Disk: Raw Device Mapping (Physical)"]
        end
        subgraph VertNet["Dedicated Networks"]
            VMXNET3["vNIC: VMXNET3 / SR-IOV"]
        end
    end

    subgraph BadPractice["SHARED KAFKA VM (Anti-Pattern)"]
        direction TB
        subgraph SharedResource["Shared Resources (Shared)"]
            vCPU_Share["vCPU: 1:4 Ratio (High Steal Time)"]
            RAM_Share["RAM: Dynamic Allocation (Ballooning)"]
        end
        subgraph BadStorage["Standard Storage"]
            LSI_Logic["Controller: LSI Logic (Default)"]
            VMDK["Disk: Standard VMDK (Thin Provisioned)"]
        end
        subgraph BadNet["Standard Networks"]
            E1000["vNIC: E1000 Emulation"]
        end
    end

Cloud VM Scenarios: Instance Type Selection Guides #

For those of us running Apache Kafka in Cloud Providers, we don’t configure hypervisors directly. Instead, we must carefully choose Instance Types provided by vendors.

Here are instance type selection recommendations at three major cloud providers:

1. Amazon Web Services (AWS EC2) #

• Recommended Instance Types: i3en or r6i / r6g series.
• Why:
  • The i3en (Storage Optimized) series is equipped with local NVMe SSD storage (instance stores) offering very high write IOPS with micro latency. Perfect for Kafka data disks.
  • The r6i (Memory Optimized) series offers large RAM-to-vCPU ratios, providing abundant page cache memory allocations.
• IMPORTANT: If using EBS (Elastic Block Store), always choose GP3 or io2 volumes and make sure our instances have EBS-Optimized status to guarantee dedicated I/O bandwidth.

2. Google Cloud Platform (GCP Compute Engine) #

• Recommended Instance Types: n2-highmem or n2d-highmem with additional Local SSDs.
• Why: The n2-highmem series provides large RAM per vCPU. Installing external Local SSDs as Kafka data directories delivers the best sequential I/O performance compared to standard Persistent Disks (PDs).

3. Microsoft Azure #

• Recommended Instance Types: Edsv4 or Lsv2 series.
• Why: The Lsv2 (Storage Optimized) series offers very large local disk throughput, while Edsv4 (Memory Optimized) is excellent for holding large page caches in RAM.

Production Virtual Machine (VM) Readiness Audit Checklist #

Do the following configuration audit steps at hypervisor and Guest OS levels before releasing Kafka VM clusters to production stages:

Summary #

• Lock Resources 100% — Always do full RAM memory and vCPU core reservations (100% Reserved) on hypervisors to eliminate CPU steal time and turn off memory ballooning.
• Use Paravirtual SCSI — Choose PVSCSI or VirtIO type virtual disk controllers to minimize hypervisor I/O translation overhead and speed up disk write throughput.
• Optimize Virtual Networks — Install VMXNET3 paravirtualized network drivers or enable SR-IOV to eliminate network jitter and maximize outbound throughput to consumers.
• Choose Memory/Storage Instances — At cloud providers, pick memory-optimized or storage-optimized instance categories equipped with local NVMe SSD disks to guarantee stable I/O latency.

← Previous: Kafka on Bare Metal Next: Kafka on Docker & Kubernetes →