Event #

In modern event streaming architectures, an event is the smallest unit of data and the fundamental fact representing a real occurrence in your business. Unlike traditional message queue systems that treat messages as temporary instructions to be deleted right away, Apache Kafka treats events as immutable historical records stored permanently. Deeply understanding the event’s anatomy, distribution mechanism, and schema management is a crucial foundation before designing a resilient, large-scale distributed system.

Basic Concept: What is an Event? #

In the Apache Kafka system, an event (often also called a record or message) records the fact that something has happened in your system or business. This fact is immutable, meaning once an event is written to the Kafka log, it cannot be changed, deleted, or modified by anyone. This immutability is very important because it guarantees the integrity of historical data that you can audit at any time without worrying about mid-stream manipulation.

As a concrete example, in an e-commerce application, real occurrences like “Customer A added item X to the shopping cart at 10:00” or “Transaction Y’s payment was successfully verified at 10:05” are written as individual events. Each event carries a time context and data payload recording the exact state at the moment it happened.

The fundamental difference between Kafka and traditional message brokers like RabbitMQ lies in how events are treated after consumption. In RabbitMQ, once a consumer receives and acknowledges a message, it’s immediately deleted from the broker’s memory to save storage space. In Kafka, events stay stored on the broker’s disk according to the retention policy you set, allowing other consumers to replay the entire historical sequence from any starting point.

Physical Anatomy and Structure of an Event #

Physically, at the binary level, an event sent by a producer application and stored by a Kafka broker is not just plain JSON or XML text. Kafka wraps the data into an orderly binary structure consisting of four main components: Key, Value, Timestamp, and Headers.

Let’s visualize the logical binary structure of an event below:

flowchart TD
    subgraph EventBiner["Kafka Event Binary Structure"]
        direction TB
        Headers["Headers <br/> (Optional metadata: Trace ID, Schema ID, Token)"]
        Timestamp["Timestamp <br/> (8 bytes: CreateTime vs LogAppendTime)"]
        Key["Key <br/> (Byte Array: Identity & Partition Route)"]
        Value["Value <br/> (Byte Array: Main Business Payload)"]
    end

1. Key #

The Key is optional data in the form of a byte array sent alongside the main payload. Although optional (it can be null), the key plays a crucial role in Kafka’s architecture. Its main function is to serve as the logical identity pointer for the data (e.g., User ID, Transaction ID, or Vehicle Registration Number) and is used by the producer’s internal algorithm to determine which partition the event should be sent to.

2. Value #

The Value is the main data payload you want to send, also stored as a byte array. This is where your primary business information lives. Because Kafka treats the value only as raw binary, you’re free to use any data format to represent this information, such as JSON, Apache Avro, Protocol Buffers (Protobuf), XML, or plain text. The Kafka broker never reads or cares about the contents of this value; it simply receives, stores, and efficiently serves it back to consumers.

3. Timestamp #

Every event in Kafka must have an 8-byte timestamp. This timestamp records when the event occurred or when it was written to the broker. There are two main timestamp types supported natively by Kafka: CreateTime (the timestamp when the producer created the event on the client side) and LogAppendTime (the timestamp when the broker received and wrote the event to local disk storage).

4. Headers #

Added since Kafka version 0.11, Headers let you attach extra metadata as key-value pairs without polluting or altering the main business data structure inside the Value. The keys are strings, while the values are stored as byte arrays. This feature is very useful for system utilities like distributed tracing, schema version management, encryption, and authentication.

The Role of the Key in Data Partitioning #

One of the biggest challenges in managing distributed systems is how to spread workloads evenly across multiple servers without losing data ordering guarantees. Kafka solves this by splitting topics into multiple partitions, and this is where the Key on an event plays a central role.

When a producer application sends an event, the producer checks whether the event has a Key or not. This decision flow determines the physical storage route of your data:

flowchart TD
    Start(["Producer Sends Event"]) --> CheckKey{"Is the Key null?"}
    CheckKey -- Yes --> RoundRobin["Sticky Partitioning / Round-Robin <br/> (Events distributed evenly across all partitions)"]
    CheckKey -- No --> Hashing["Compute Hash of Key <br/> (Formula: MurmurHash2(Key) % Number of Partitions)"]
    Hashing --> FixedPartition["Send to the Hash Result Partition <br/> (Guarantee: Same key goes to the same partition)"]

The Murmur2 Hashing Algorithm #

If the Key is not null, the producer by default uses the MurmurHash2 hashing algorithm to turn the key value into a 32-bit integer representation. Once the hash value is obtained, the producer performs a modulo operation against the total number of active partitions available on that topic:

$$\text{Partition} = \text{MurmurHash2}(\text{Key}) \pmod{\text{Number of Partitions}}$$

Through the mathematical formula above, as long as the number of partitions on the topic doesn’t change, the same Key value will always produce the exact same partition index. This provides an absolute guarantee that all events related to a specific business entity (for example, all transaction history from User_123) will always be written to the same partition and read in order by the same consumer.

Sticky Partitioning for Null Keys #

If you send an event with a null Key, the producer can’t use hashing. In early Kafka versions, the producer would distribute data across all partitions one by one in a round-robin fashion. However, this method is inefficient because it generates many small network packets continuously sent to different brokers.

Since Kafka 2.4, the Sticky Partitioner algorithm was introduced. When the Key is null, the producer picks one partition at random and sends all subsequent events to that same partition until the accumulator batch size (batch.size) is reached or the wait limit (linger.ms) expires. Once the batch is sent to the broker, the producer picks a new random partition for the next batch. This strategy greatly improves network efficiency and minimizes latency without sacrificing long-term load distribution.

The Danger of Key Skewness (Hot Partitions) #

Although using a Key is very helpful for preserving message order, you must be wary of Key Skewness (lopsided partitions). This happens when you choose a key whose real-world distribution is uneven.

For example, if you choose a key based on the e-commerce transaction’s Country_Of_Origin, and 90% of your customers come from Indonesia (ID), then the partition hashed from the ID key will receive 90% of your site’s total data, while the other partitions sit idle. As a result, the broker server managing that hot partition suffers CPU and disk I/O overload, triggering system-wide bottlenecks. To prevent this, always choose Keys with high entropy and random distribution, such as unique User IDs (user_id) or transaction UUIDs.

Timestamps in Kafka #

The 8-byte timestamp in every Kafka event plays an important role in broker operations, stream processing, and disk storage management. Kafka provides a topic-level configuration (message.timestamp.type) to determine how timestamps are managed in the cluster:

1. CreateTime (Producer Side) #

By default, the timestamp type is CreateTime. When your producer application creates an event object, the producer injects the server’s local time into the event metadata. If the producer and broker servers have poor time synchronization via NTP (Network Time Protocol), this timestamp may not reflect the actual time when the broker received the data.

2. LogAppendTime (Broker Side) #

If the topic configuration is set to LogAppendTime, the Kafka broker ignores any timestamp sent by the producer. Right after the broker receives the event and before writing it to the local log segment on disk, the broker injects the broker server’s time as the official timestamp. This method guarantees that all timestamps within a partition are monotonically increasing.

Impact on Retention Policy and Windowing #

Choosing a timestamp type has significant practical consequences for your system architecture:

• Data Retention: Time-based retention policies in Kafka (e.g., deleting data after 7 days) are calculated based on the timestamp stored in the event. If you use CreateTime and the producer accidentally sends data with last year’s timestamp due to a system bug, Kafka will immediately delete that data right after it’s written because it’s considered already expired.
• Windowing in Kafka Streams: When doing real-time event streaming processing (e.g., calculating total transactions per 5 minutes), consumer applications group data based on the event timestamp. Using CreateTime reflects the real business occurrence time (event time), while LogAppendTime reflects the system’s data recording time (processing time).

Headers Metadata for Tracing and Audit #

Headers in Kafka provide a very flexible low-level metadata structure. This feature is designed similarly to HTTP protocol headers, where metadata is sent as part of the event envelope, separate from the main payload content.

Let’s study some real use cases of Headers in the industry:

1. Distributed Tracing #

In microservices architectures, a single user business flow (for example, pressing the buy button) can trigger dozens of asynchronous API calls between services connected by Kafka. To trace this flow, you need a unique trace ID (Trace ID and Span ID) based on the OpenTelemetry standard.

By inserting the Trace ID into Kafka Headers, every microservice that reads from and writes back to Kafka can propagate the tracing context. You can see the complete service call flow graph in monitoring systems like Jaeger or Zipkin without forcing every microservice developer team to modify their business JSON payload structure.

2. Schema Version Management (Schema Registry Interaction) #

When producers send binary data like Apache Avro, the producer needs to tell consumers which schema version was used to encode the data. The producer inserts a unique Schema ID (4 bytes) into the event header. When a consumer reads binary data from Kafka, it first reads the Schema ID from the header, downloads the matching schema from the Schema Registry server, then safely and quickly decodes the main payload.

The Importance of Schema Registry and Data Contracts #

One fatal mistake often made by developers new to Kafka is sending payload data in raw JSON format without strict schema rules. Because Kafka treats payloads as raw binary, producers are free to send anything. Without a clear data contract, small changes on the producer side can break the entire downstream consumer system.

// ANTI-PATTERN: Producer sends free-form JSON data without schema control
{
  "id_transaksi": 9012,
  "nominal": 150000        // Consumer depends on the "nominal" field being an integer
}

// Producer ships a code update without coordination:
{
  "transaction_id": 9012,  // Field name changed
  "nominal": "150,000"     // Data type changed to a string with thousands separator
}
// RESULT: Every consumer application crashes instantly because it fails to parse the data!

To solve this large-scale coordination problem, you must use the Confluent Schema Registry as the data contract manager between producers and consumers. The highly recommended data formats for the Kafka ecosystem are Apache Avro or Protocol Buffers (Protobuf).

Below is an illustration of how the Schema Registry acts as a data traffic police, preventing corrupt data from entering your Kafka cluster:

sequenceDiagram
    participant P as Producer
    participant SR as Schema Registry
    participant K as Kafka Broker
    participant C as Consumer

    P->>SR: 1. Register / Validate New Schema
    alt Schema Compatible
        SR-->>P: return Schema ID (e.g. ID 4)
        P->>K: 2. Send Binary Payload + Schema ID in Header
        K->>C: 3. Pull Binary Event + Schema ID
        C->>SR: 4. Fetch Detailed Schema for ID 4
        SR-->>C: return Avro/Protobuf Schema
        Note over C: Successfully decoded binary into data object!
    else Schema Violates Compatibility Rules
        SR-->>P: Throw Exception (Registration Rejected!)
        Note over P: Sending to Kafka is automatically cancelled
    end

Types of Schema Compatibility Rules #

Using the Schema Registry, you can set schema evolution rules that are safe for your consumers:

• BACKWARD Compatibility (Default Recommendation): Consumers with the new schema can read old data written by producers with the old schema. This lets you safely update consumer applications first.
• FORWARD Compatibility: Consumers with the old schema can read new data written by producers with the new schema. This is useful when you want to update producer applications first without breaking old consumer functionality.
• FULL Compatibility: Schema changes are compatible both backward and forward. You’re free to update producers or consumers in any order without parsing failure risks.

Let’s compare the three popular schema formats commonly used in the Apache Kafka ecosystem:

Null Key vs Keyed Event Usage Scenarios #

When designing an event-based system, the decision of whether to include a Key or leave it null directly impacts data ordering and load distribution performance. Use the decision guide table below to determine the right strategy for your system:

CHOOSE Keyed Events (Key NOT Null) if:
  ✓ Logical data ordering by a specific entity must be preserved (e.g., transaction order per Account ID).
  ✓ You adopt the Event Sourcing or CQRS architecture pattern.
  ✓ You want to perform data joins or stateful aggregation using Kafka Streams (KTable).
  ✓ You use the Log Compaction strategy to keep the last event of each unique key.

CHOOSE Non-Keyed Events (Key = Null) if:
  ✗ Global ordering across entities doesn't matter to your business (e.g., random system server log collection).
  ✗ Maximum possible data delivery throughput is your top priority.
  ✗ You want data load distribution spread absolutely 100% evenly across all partitions and brokers.

Implementation Code: Sending Events Correctly and Safely #

Let’s study the producer code example below using Python to demonstrate sending events with proper schemas and keys:

# ANTI-PATTERN: Sending raw JSON messages without a key and without error handling
# This causes loss of entity data ordering guarantees and makes failure tracking difficult.
def kirim_transaksi_salah(producer, data_transaksi):
    import json
    payload = json.dumps(data_transaksi).encode('utf-8')
    # Sending without a key and without a callback (fire-and-forget)
    producer.send('topik-transaksi', value=payload)

# CORRECT: Sending events with the right Key to guarantee per-entity data ordering
# Equipped with asynchronous callback handling to ensure successful delivery to the broker.
def kirim_transaksi_benar(producer, data_transaksi):
    import json
    
    # Account ID used as the Key to guarantee all transactions of this account land in the same partition
    key_biner = str(data_transaksi['account_id']).encode('utf-8')
    payload_biner = json.dumps(data_transaksi).encode('utf-8')
    
    # Callback handling for send results (asynchronous)
    def on_send_success(record_metadata):
        # Include partition and offset info for our application's audit logs
        print(f"✓ Event sent to topic {record_metadata.topic} "
              f"partition [{record_metadata.partition}] with offset {record_metadata.offset}")
              
    def on_send_error(ex):
        # Error logger to trigger alert systems or data resends
        print(f"✗ Failed to send event to Kafka cluster: {ex}")
        # DON'T: ignore this error in production, do logging/retry
        simpan_ke_antrean_lokal_retry(data_transaksi)

    # Sending with a Key and callback handling
    producer.send(
        topic='topik-transaksi',
        key=key_biner,
        value=payload_biner
    ).add_callback(on_send_success).add_errback(on_send_error)

Summary #

• Event Definition — An event is an immutable record of a business fact, stored persistently on the Kafka broker’s disk, and replayable at any time by various independent consumers.
• Event Components — The logical structure of a binary event consists of a Key (optional, for partition routing), Value (main business payload), Timestamp (8-byte timestamp), and Headers (distributed tracing metadata).
• Partition Algorithm — Events with a non-null Key are consistently mapped to the same partition using the hash modulo formula MurmurHash2(Key) % Number of Partitions. Events without a Key are distributed efficiently using the Sticky Partitioning strategy.
• Key Skewness Danger — Avoid Keys with low value diversity (like country or gender) because they can cause data to pile up on a single partition, slowing overall system performance.
• Timestamps — Kafka supports CreateTime (producer-created time) and LogAppendTime (broker recording time) timestamps, which affect data retention policies and windowing logic.
• Schema Data Contract — Never send free-form JSON data without a schema in large production environments. Use a Schema Registry with Apache Avro or Protobuf to ensure safe schema compatibility across application versions.