📨 Section 15 · Question #17

What are Leader and Follower Replicas

In Apache Kafka's partitioned replication model, each topic partition has a designated replica set distributed across different physical brokers. Within this replica group, resp...

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

In Apache Kafka’s partitioned replication model, each topic partition has a designated replica set distributed across different physical brokers. Within this replica group, responsibilities are strictly divided into two distinct roles:

  1. Leader Replica: Exactly one replica per partition acts as the primary authoritative coordinator. It serves as the sole entry point for all client write operations (Produce) and handles read operations (Consume) by default. The Leader maintains the In-Sync Replicas (ISR) set and calculates the High Watermark (HW).
  2. Follower Replicas: The remaining $N - 1$ replicas are passive mirrors. They never accept write operations from clients. Their sole responsibility is to continuously pull new log segments from the Leader via internal FetchRequest RPCs, replicating its commit log byte-for-byte. If the Leader crashes, the cluster controller instantly promotes an up-to-date follower to become the new Leader.

Real-World Analogy

Imagine an orchestra conductor and the first violinists:

  • Leader (Conductor): Receives the newly composed sheet music from the composer (producer), sets the tempo, and directs the live performance (handles active I/O).
  • Followers (Violinists): Closely observe the conductor’s every gesture, instantly playing each note in perfect synchronization on their instruments.
  • If the conductor suddenly collapses on stage, the first chair violinist immediately picks up the baton and continues the symphony from the exact same measure without missing a beat.

Inspecting Replica Roles

# Describe partition replica distribution for the orders topic
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders

Sample Output:

Topic: orders   PartitionCount: 3   ReplicationFactor: 3
  Topic: orders  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
  Topic: orders  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3,1
  Topic: orders  Partition: 2  Leader: 3  Replicas: 3,1,2  Isr: 3,1,2

In this topology, Broker 1 is the active Leader for Partition 0, while Brokers 2 and 3 serve as its passive followers.

🟡 Middle Level

Follower Replication Mechanics: ReplicaFetcherThread

Follower replicas ingest data from the Leader using an internal replication engine structured almost identically to a standard consumer:

  1. Each broker runs a pool of background threads named ReplicaFetcherThread (configured via num.replica.fetchers).
  2. The fetcher thread issues a FetchRequest to the partition Leader, specifying its current local Log End Offset (LEO).
  3. The Leader reads messages from its local segment files (leveraging the Linux OS PageCache and kernel zero-copy sendfile system call) and returns the batch to the follower.
  4. The follower writes the batch to its own local append-only log segment and immediately sends the next FetchRequest, implicitly acknowledging receipt and updating the Leader on its progress.
                    Internal Replication Fetch Loop
[ Producer ] ───(Produce)───► [ Broker 1 (Leader) ]
                                      ▲
                              FetchRequest(LEO=105)
                                      │
                              FetchResponse(105..110)
                                      ▼
                               [ Broker 2 (Follower) ]
                               (ReplicaFetcherThread)

Preferred Leader and Automatic Rebalancing

In topic metadata, the assigned replica list is strictly ordered: Replicas: [2, 1, 3].

  • The first broker in this list (2) is designated as the Preferred Leader.
  • When the topic is initially provisioned, Kafka intentionally distributes preferred leaders evenly across all brokers to ensure balanced CPU and network I/O.
  • If Broker 2 reboots, Broker 1 temporarily assumes leadership. However, once Broker 2 recovers and rejoins the ISR, the background thread governed by auto.leader.rebalance.enable = true automatically restores leadership to Broker 2, preventing load imbalances from accumulating.

🔴 Senior Level

Evolution of Lag Detection: Why replica.lag.max.messages Was Removed

In early versions of Kafka (prior to 0.9), follower desynchronization was evaluated using message count:

  • If a follower lagged behind the Leader by more than $N$ messages, it was evicted from the ISR.
  • The Architectural Flaw: During sudden bursty traffic spikes, a producer could publish 50,000 messages in 100 milliseconds. Healthy followers, processing data at normal disk speeds, would momentarily fall 50,000 messages behind and get dropped from the ISR, triggering false alarms, alert storms, and unneeded rebalances.
  • Modern Solution: The message count metric was completely abolished. Kafka strictly relies on replica.lag.time.max.ms (default: 30,000 ms / 30 seconds). As long as a follower issues a valid FetchRequest within 30 seconds, it remains an active member of the ISR regardless of how many bytes or records it is currently lagging behind.

Follower Fetching: KIP-392 and Multi-AZ Cloud Topologies

By default, clients communicate exclusively with the Leader. However, in modern cloud deployments (e.g., AWS multi-AZ, GCP regions), cross-AZ network traffic introduces high operational costs and latency overhead.

Kafka 2.4+ introduced Follower Fetching (KIP-392):

  1. Brokers declare their rack/zone: broker.rack=us-east-1a.
  2. Consumers declare their physical rack/zone: client.rack=us-east-1a.
  3. The cluster activates rack-aware routing: 
    replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector
  4. If an In-Sync Follower resides in the same cloud AZ as the consumer, the consumer fetches records directly from that local follower, eliminating cross-AZ egress costs. The follower strictly serves records up to its local High Watermark, guaranteeing that uncommitted data is never exposed.

4 Tricky Questions

1. How does Kafka guard against Split-Brain (“Zombie Leader”) scenarios when an old leader recovers after being replaced?

Answer:

  • Suppose Leader $L_1$ suffers a 40-second GC pause or network partition. The cluster controller detects the heartbeat timeout, promotes follower $L_2$ to Leader, and increments the Leader Epoch counter from $N$ to $N+1$.
  • When $L_1$ recovers from its pause, it still erroneously believes it is the legitimate Leader for epoch $N$.
  • When a producer sends a write or when $L_1$ attempts to communicate with the rest of the cluster, brokers and clients inspect the epoch metadata attached to the request.
  • Detecting a stale epoch ($N < N+1$), the request is immediately rejected with a FencedLeaderEpochException. The zombie leader is forcibly fenced, drops its leadership status, invalidates its cached metadata, and demotes itself to a follower.

2. Can a consumer encounter “time travel” (reading stale or inverted offsets) when reading from a Follower replica under KIP-392?

Answer:

  • No. A consumer tracks its current read progress monotonically.
  • Follower replicas are only permitted to serve records up to their local High Watermark, which represents data already acknowledged across the ISR.
  • If a consumer reads up to offset 500 from the Leader and subsequently switches to a local Follower whose local High Watermark is temporarily at 495 due to replication lag, the Follower does not return stale data or fail silently.
  • Instead, the follower holds the FetchRequest until its local log reaches offset 500 (within fetch.max.wait.ms), or returns an offset-out-of-range instruction prompting the consumer to wait. Monotonic offset progression is strictly preserved.

3. Why is a partition briefly unavailable during a Leader crash if follower replicas already possess all committed records?

Answer:

  • Failover is near-instantaneous, but distributed consensus requires a non-zero coordination window:
    1. Failure Detection: The cluster controller must detect the leader’s demise via heartbeat timeout (typically 3–5 seconds in ZooKeeper/KRaft).
    2. Metadata Mutation: The controller writes the new Leader assignment and incremented Leader Epoch into the cluster metadata log (KRaft Raft quorum log).
    3. Metadata Propagation: The controller disseminates updated metadata (UpdateMetadataRequest) to all active brokers in the cluster.
    4. Client Re-Routing: Producers and consumers attempting to reach the dead leader receive NotLeaderOrFollowerException, invalidate their local metadata caches, and request fresh cluster topology from bootstrap servers.
  • This end-to-end failover sequence typically completes within a few hundred milliseconds to a couple of seconds.

4. What happens if all followers fall out of ISR, and then the Leader crashes?

Answer:

  • The cluster faces the classic CAP theorem trade-off:
    • If unclean.leader.election.enable = false (Default / Production standard): The partition becomes completely unavailable for both reads and writes. The cluster refuses to promote an out-of-sync follower and waits indefinitely for the crashed Leader to recover. Consistency is preserved at the total sacrifice of Availability.
    • If unclean.leader.election.enable = true: The controller elects an out-of-sync lagging follower as the new Leader. The system recovers availability immediately, but any messages committed to the old Leader that never reached this follower are permanently lost, and existing offsets will be overwritten.

🎯 Interview Cheat Sheet

Core Concepts

  • Leader: The single authoritative replica per partition that coordinates client writes, default reads, ISR membership, and High Watermark calculation.
  • Follower: A passive replica that mirrors the Leader’s log via background ReplicaFetcherThread workers.
  • Preferred Leader: The first broker in the partition’s assigned replica list; Kafka actively attempts to maintain leadership on this node for uniform cluster load.
  • Zombie Fencing: The Leader Epoch mechanism prevents split-brain writes by rejecting outdated leaders with FencedLeaderEpochException.
  • KIP-392 Follower Fetching: Allows consumers to read from local AZ followers using client.rack to eliminate cloud egress bandwidth costs.

Leader vs. Follower Comparison

| Capability / Characteristic | Leader Replica | Follower Replica | | :— | :— | :— | | Accepts Client Writes (Produce) | ✅ Exclusively | ❌ Rejects with NotLeaderOrFollowerException | | Serves Client Reads (Consume) | ✅ By default | ⚠️ Only when configured via KIP-392 (client.rack) | | Data Ingestion Method | Direct append from producers | Pulls batches via FetchRequest to Leader | | Manages ISR and High Watermark | ✅ Calculates and publishes | ❌ Applies updates passively | | Eligible for Failover Promotion | N/A (Already Leader) | ✅ If currently present in active ISR |

Red Flags to Avoid

  • ❌ “Followers accept writes from producers to balance cluster write throughput.” (Only the Leader accepts writes; followers replicate sequentially).
  • ❌ “Followers are evicted from the ISR if they fall behind by more than 100 messages.” (Message count lag was deprecated; eviction is strictly time-based via replica.lag.time.max.ms).
  • ❌ “When a Leader dies, uncommitted messages in its memory are lost.” (Clients only receive success confirmations for records committed up to the High Watermark in the ISR).