📨 Section 15 · Question #12

What is Offset in Kafka

An Offset in Apache Kafka is a sequential, monotonically increasing 64-bit signed integer (long) assigned by the partition leader to every incoming message as it is appended to...

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

An Offset in Apache Kafka is a sequential, monotonically increasing 64-bit signed integer (long) assigned by the partition leader to every incoming message as it is appended to disk. It serves as the immutable, sequential logical address of a record within a specific partition.

Core Properties:

  • Partition-Local Scope: An offset is unique only within its individual partition. Partition 0 and Partition 1 both contain distinct records at offset 42.
  • Strict Monotonicity: Offsets increment predictably (0, 1, 2, 3...) and never decrement or re-sequence, even when older records are purged by retention policies.
  • Consumer Progress Tracking: Consumers maintain their reading position by persisting their last processed offset back to the cluster (Committed Offset), enabling failover recovery without missing records.

Real-World Analogy

Think of a topic as a multi-volume encyclopedia with bookmarks:

  • Topic: The entire encyclopedia set.
  • Partition: An individual physical volume (Volume 1, Volume 2, Volume 3).
  • Offset: The page number inside a specific volume. Every volume has a page 10, but each contains entirely different text.
  • Consumer Group: A researcher reading the encyclopedia. When the researcher finishes page 15 of Volume 2, they place a colored bookmark (Committed Offset) on page 15. When returning the next day, they resume reading from page 16.

Partition Offset Landmarks

Partition 0 Log:
┌───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐
│ Offset 0  │ Offset 1  │ Offset 2  │ Offset 3  │ Offset 4  │ Offset 5  │ ... (New Appends)
└───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘
      ▲                       ▲                       ▲           ▲
      │                       │                       │           │
Log Start Offset       Committed Offset        High Watermark    Log End Offset (LEO)
(Oldest available)   (Consumer progress)    (Safe for consumers) (Next broker write)

🟡 Middle Level

The 5 Critical Partition Offset Markers

To diagnose lag and build reliable consumers, an engineer must distinguish five core offset boundaries:

  1. Log Start Offset: The offset of the oldest unpurged record currently retained on disk. Records prior to this mark were permanently purged by retention cleanup (retention.ms or retention.bytes).
  2. Current Position: The offset of the next record that will be returned to the application on the subsequent consumer.poll() execution.
  3. Committed Offset: The latest offset successfully acknowledged and persisted into the __consumer_offsets topic by the consumer group.
  4. High Watermark (HW): The offset up to which records have been acknowledged as replicated across all In-Sync Replicas (ISR). Standard consumers are restricted to reading only up to the High Watermark!
  5. Log End Offset (LEO): The offset of the next record to be written into the leader’s local partition log. In a live cluster, $\text{LEO} \ge \text{HW}$.
\[\text{Consumer Lag} = \text{Log End Offset (LEO)} - \text{Committed Offset}\]

Programmatic Offset Management in Java

Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "ledger-service");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    TopicPartition partition = new TopicPartition("payments", 0);
    consumer.assign(List.of(partition));

    // 1. Inspect current fetch position:
    long nextOffsetToRead = consumer.position(partition);
    System.out.println("Next fetch offset: " + nextOffsetToRead);

    // 2. Programmatic seek (Repositioning / Time Travel):
    consumer.seek(partition, 1500L); // Jump directly to offset 1500
    // consumer.seekToBeginning(List.of(partition)); // Replay from oldest available record
    // consumer.seekToEnd(List.of(partition));       // Fast-forward to latest head (skip backlog)

    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    for (ConsumerRecord<String, String> record : records) {
        processPayment(record.value());

        // 3. Commit the NEXT offset to process (record.offset() + 1):
        consumer.commitSync(Map.of(
            partition, new OffsetAndMetadata(record.offset() + 1, "processed-v1")
        ));
    }
}

🔴 Senior Level

Offset Persistence: The __consumer_offsets System Topic

Prior to Kafka 0.9, consumer offsets were committed to Apache ZooKeeper, causing high write contention and cascading cluster outages. Modern Kafka persists all group offsets into the internal compacted system topic __consumer_offsets:

  • Topology: 50 partitions with a replication factor of 3 by default.
  • Log Compaction: Retains only the most recent committed offset record for each unique key.
  • Record Schema:
    • Key: [group.id, topic, partition]
    • Value: [offset, leaderEpoch, metadata, commitTimestamp]

The Group Coordinator broker responsible for managing a specific consumer group is calculated mathematically:

\[\text{partition} = |\text{groupId.hashCode()}| \pmod{50}\]

The broker hosting the Leader replica of that __consumer_offsets partition acts as the Group Coordinator.

Low-Level Storage Engine: Sparse Indexing (.index)

Kafka locates records in multi-terabyte logs within microseconds using Memory-Mapped Sparse Indexes: Each partition segment on disk consists of three files:

  1. 00000000000000000000.log: Sequential raw serialized records.
  2. 00000000000000000000.index: Sparse index mapping logical offsets to physical byte positions.
  3. 00000000000000000000.timeindex: Time-based index mapping timestamps to logical offsets.

Memory-Mapped Binary Search:

  • The .index file does not index every record. It appends an index entry approximately every 4 KB of written data (index.interval.bytes).
  • Each entry occupies 8 bytes: [Relative Offset (4 bytes) -> Physical Log Position (4 bytes)].
  • The broker memory-maps (mmap) the .index file directly into the OS Page Cache. It performs an $O(\log N)$ binary search on the index to find the closest preceding byte offset, followed by a short, sequential disk read of a few kilobytes in the .log file.

4 Tricky Questions

1. What is the fundamental difference between Log End Offset (LEO) and High Watermark (HW), and why are consumers strictly forbidden from reading up to LEO?

Answer:

  • Log End Offset (LEO) represents the offset of the next record to be written into the partition leader’s local storage. The leader has received and written this record to disk/page cache, but follower replicas in the ISR have not yet replicated it.
  • High Watermark (HW) represents the highest offset that has been confirmed replicated across all In-Sync Replicas (ISR).
  • If a consumer were permitted to read dirty records up to LEO, and the leader node suffered catastrophic power loss before replicating those offsets to followers, a new leader elected from the remaining ISR would not possess those records. The consumer would have processed phantom records that permanently disappeared from the cluster, causing unrecoverable data corruption.
  • Restricting consumption strictly up to the HW guarantees monotonic consistency across leader elections.

2. How did the Leader Epoch mechanism (KIP-101) solve the historical “Silent Data Loss” bug during replica truncation?

Answer:

  • Prior to Kafka 0.11, follower replica synchronization relied exclusively on the High Watermark.
  • The Classic Bug Scenario: Follower B lagged behind and restarted. Upon booting, B truncated its log to its last known local HW. If Leader A crashed before B caught up, B was elected leader with a truncated log. When Node A revived, it also truncated its log down to B’s truncated HW. Client-acknowledged records were silently erased from the cluster without throwing errors.
  • KIP-101 Leader Epoch Solution: Kafka stamps every message with a monotonically incremented Leader Epoch counter indicating which leader wrote the record. When a follower rejoins, it sends an OffsetForLeaderEpochRequest to the current leader. Instead of blindly truncating to its old local HW, the follower truncates only to the point where the new leader’s epoch began, mathematically eliminating silent data loss and log divergence.

3. What happens if a consumer commits offset 500, but due to extended service downtime, offsets 0–800 are pruned by the log retention cleaner, and how does auto.offset.reset dictate behavior?

Answer:

  • When the consumer restarts and requests records starting at offset 500, the broker observes that the partition’s Log Start Offset has advanced to 801.
  • The broker returns an OffsetOutOfRangeException because offset 500 no longer exists on disk.
  • How the client recovers depends entirely on its auto.offset.reset configuration:
    • earliest: Automatically resets reading position to the earliest available offset on disk (801). The consumer resumes processing; records 500–800 are permanently lost to this consumer.
    • latest: Skips the entire backlog and jumps directly to the head of the log (Log End Offset), consuming only newly published records.
    • none: Throws a fatal exception into the application thread, halting consumption until an operator manually resets the group’s offset using kafka-consumer-groups.sh.

4. Can an offset in a Kafka partition experience integer overflow, and what happens mathematically if it reaches its maximum value?

Answer:

  • An offset in Kafka is a signed 64-bit integer (long), with a theoretical ceiling of $2^{63} - 1 = 9,223,372,036,854,775,807$.

  • Even if a high-throughput producer pushes 1,000,000 messages per second continuously into a single partition without pausing:

    \[\frac{9.22 \times 10^{18}}{10^6 \times 3600 \times 24 \times 365} \approx 292,471 \text{ years}\]
  • Practical offset overflow in real-world clusters is mathematically impossible. In Kafka’s specification, reaching the maximum value would prevent appending new records to that specific partition, requiring the topic to be recreated or truncated.

🎯 Interview Cheat Sheet

Core Concepts at a Glance

  • Definition: Monotonically increasing 64-bit integer identifier scoped strictly to an individual partition.
  • Partition Scoping: Offsets are independent per partition; never assume global ordering across a topic.
  • High Watermark (HW): The boundary of data replicated across all ISR replicas; consumers cannot read beyond HW.
  • Consumer Lag: $\text{LEO} - \text{Committed Offset}$. The primary operational metric for consumer health.
  • Storage Architecture: Tracked in __consumer_offsets (50 partitions, compacted).
  • Sparse Indexing: Memory-mapped .index files allow $O(1)$ disk offset lookups via binary search.
  • Leader Epoch: Prevents data divergence and log truncation data loss during broker failovers.

Partition Offset Landmark Hierarchy

Landmark Definition Controlled By
Log Start Offset Oldest available unpurged record Retention Cleaner
Committed Offset Highest acknowledged position saved Consumer Group
Current Position Next offset to be fetched by poll() Application Client
High Watermark Highest offset replicated across all ISR Broker ISR Quorum
Log End Offset Next offset to be written to disk Partition Leader

Red Flags to Avoid

  • ❌ “Kafka offsets provide global message ordering across all topic partitions.” (Offsets are strictly partition-local).
  • ❌ “Offsets are stored in Apache ZooKeeper in modern Kafka clusters.” (Stored in __consumer_offsets since Kafka 0.9).
  • ❌ “Consumers read uncommitted records up to Log End Offset immediately as producers publish.” (Consumers are strictly bounded by the High Watermark).
  • ❌ “When older records are deleted by retention, Kafka shifts remaining offsets back down to zero.” (Offsets are immutable; retention simply advances the Log Start Offset).