📨 Section 15 · Question #18

What is ISR (In-Sync Replicas)

Key characteristics of ISR:

📨 Kafka 📄 Source Markdown

🟢 Junior Level

30-Second Summary

ISR (In-Sync Replicas) is the dynamic subset of a partition’s configured replicas that are currently alive, reachable, and fully caught up with the partition Leader’s commit log.

Key characteristics of ISR:

  • Leader Inclusion: The Leader is always an inherent, mandatory member of its own ISR set.
  • Failover Eligibility: When the active Leader crashes, the cluster elects a new Leader exclusively from the active ISR set. This fundamental invariant guarantees zero data loss for acknowledged messages.
  • Dynamic Membership: If a follower replica falls behind due to network degradation, heavy GC pauses, or disk bottlenecks, it is evicted from the ISR (Shrink). As soon as the follower catches up with the Leader’s log, it is automatically reinstated (Expand).

Real-World Analogy

Imagine a corporate board of directors:

  • All board members must review and sign resolution protocols within 30 minutes of publication.
  • Those who have signed every recent resolution constitute the Active Quorum (ISR).
  • If the CEO (Leader) suddenly departs, the interim CEO can only be chosen from the active quorum members because they possess complete context on all recent company decisions. A director who skipped past meetings is strictly disqualified from leadership.

Inspecting ISR Status via CLI

# Check topic partition layout and active ISR membership
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments

Sample Output:

Topic: payments  Partition: 0  Leader: 1  Replicas: 1,2,3  Isr: 1,2,3
Topic: payments  Partition: 1  Leader: 2  Replicas: 2,3,1  Isr: 2,3

In Partition 1, Broker 1 has dropped out of the ISR: Brokers 2 and 3 are in-sync, while Broker 1 is marked as lagging (under-replicated).

🟡 Middle Level

ISR Lifecycle: Shrink and Expand Mechanics

The active Leader broker continuously monitors follower progress and directly administers ISR membership:

  1. Synchronization Criterion (replica.lag.time.max.ms):
    • Defaults to 30,000 ms (30 seconds).
    • The Leader monitors the timestamp of the last successful FetchRequest from each follower that caught up to the Leader’s Log End Offset (LEO).
    • If a follower fails to send a fetch request or fails to catch up within 30 seconds, the Leader triggers an ISR Shrink.
  2. ISR Shrink:
    • The Leader removes the lagging follower from its local in-memory ISR set.
    • The Leader issues an AlterPartitionRequest to the cluster controller (KRaft Quorum) to record the new ISR membership in cluster metadata.
  3. ISR Expand:
    • The lagging follower continues pulling messages in the background.
    • Once the follower’s local LEO reaches the Leader’s current LEO, the Leader acknowledges that the follower has fully caught up and submits an ISR expansion request to the controller.
[ Leader (LEO=100) ]
        │
        ├──(Fetch LEO=100 every 500ms)──► [ Broker 2 ]  -> In ISR ✅
        │
        └──(Fetch LEO=40, lag > 30s)────► [ Broker 3 ]  -> ISR Shrink ❌ (Evicted)

Quorum Interaction: ISR, acks=all, and min.insync.replicas

  • acks = all (or -1): Instructs the Leader to wait for acknowledgments from all current members of the active ISR set before confirming the write, not the static replication.factor.
  • min.insync.replicas: Enforces an absolute minimum ISR size threshold required to accept client writes:
Active ISR Size min.insync.replicas Producer Write Outcome (acks=all) Durability Status
3 2 ✅ Accepted and acknowledged across all 3 nodes Zero data loss risk
2 2 ✅ Accepted and acknowledged across surviving 2 nodes Zero data loss risk
1 2 ❌ Rejected by broker: NotEnoughReplicasException Write blocked; data protected

🔴 Senior Level

Internal Purgatory Mechanics and High Watermark Advancement

When a producer submits a write batch with acks = all:

  1. The Leader appends the batch to its local append-only log segment file on disk.
  2. The Leader places the pending request into an internal tracking structure called the DelayedProduce Purgatory (a specialized hierarchical timing-wheel wait queue).
  3. As follower replicas in the ISR submit FetchRequest RPCs containing offsets covering the pending batch, the Leader increments the acknowledgment counter.
  4. Once all active members of the current ISR acknowledge the batch:
    • The Leader advances its High Watermark (HW) pointer.
    • The request is completed and removed from the Purgatory.
    • The Leader returns RecordMetadata (including assigned offset and timestamp) to the producer client.

Architectural Evolution: ZooKeeper vs. KRaft

  • ZooKeeper Era (Kafka < 3.3):
    • Every single ISR modification forced the Leader broker to execute a synchronous write into the ZooKeeper znode: /brokers/topics/<topic>/partitions/<partition>/state.
    • Under widespread network instability or broker rolling restarts, thousands of partition Leaders simultaneously hammered ZooKeeper with znode updates, creating severe ZK ensemble write saturation, session timeouts, and cascading cluster blackouts.
  • KRaft Era (Kafka 3.3+):
    • ISR modifications are dispatched as an AlterPartitionRequest directly to the active KRaft Quorum Controller.
    • The controller serializes ISR state changes directly into the internal @metadata Raft log topic as compact metadata batches, reducing update latency to sub-milliseconds and scaling to hundreds of thousands of partitions without external dependencies.

4 Tricky Questions

1. Can a partition Leader broker ever evict itself from its own ISR set?

Answer:

  • No. An active Leader can never programmatically evict itself from the ISR. By definition, the Leader’s broker ID is a permanent and mandatory element of the ISR set.
  • If a Leader broker experiences severe hardware stalls (such as a disk lockup during an fsync call or kernel freeze), it does not shrink the ISR; rather, it stops sending background heartbeats to the cluster controller.
  • Upon heartbeat expiration, the controller declares the entire broker dead, strips its leadership role, and elects a new Leader from the remaining members of the ISR. The dead broker is then removed from cluster metadata entirely.

2. If RF = 3 and min.insync.replicas = 2, but the ISR size drops to 1, can consumers continue reading messages?

Answer:

  • Yes, consumers can read normally. The min.insync.replicas configuration enforces constraints strictly on write operations (ProduceRequest).
  • Read operations (FetchRequest from consumers) are completely unaffected. Consumers continue reading all messages up to the current High Watermark.
  • The partition effectively shifts into a Read-Only state: incoming writes are safely rejected to prevent un-replicated commits, but all existing committed data remains accessible with zero disruption to consumer pipelines.

3. Why is ISR Flapping dangerous to cluster health, and how do you remediate it?

Answer:

  • The Danger of Flapping: If a follower hovers right around the synchronization threshold (e.g., long GC pauses of 28–32 seconds against a 30-second replica.lag.time.max.ms), it repeatedly triggers Shrink $\leftrightarrow$ Expand cycles.
  • Every state transition forces metadata writes to the controller and generates cluster-wide MetadataUpdate broadcasts to all brokers and clients.
  • This creates heavy control-plane CPU spikes, network overhead, and client metadata cache invalidation storms.
  • Remediation:
    • Monitor IsrShrinksPerSec via JMX.
    • Tune JVM garbage collection (e.g., migrate to G1GC/ZGC with bounded pause targets).
    • Increase network and fetch batch buffers (replica.fetch.max.bytes).
    • Increase replica.lag.time.max.ms from 30s to 45s–60s in high-latency or shared cloud environments.

4. What happens to a pending producer (acks = all) if a follower hangs and is dropped from ISR during active replication?

Answer:

  • The Leader does not force the producer to hang indefinitely until request.timeout.ms expires.
  • As soon as the Leader detects that the stalled follower has breached replica.lag.time.max.ms, it evicts that follower from the ISR set.
  • Upon shrinking the ISR, the Leader immediately re-evaluates pending requests inside the DelayedProduce Purgatory against the newly updated, smaller ISR quorum.
  • If the remaining ISR set satisfies min.insync.replicas, the Leader advances the High Watermark and immediately sends a successful response back to the waiting producer without further delay.

🎯 Interview Cheat Sheet

Core Concepts

  • Definition: The dynamic set of partition replicas synchronized with the Leader within replica.lag.time.max.ms (default: 30s).
  • Leader Invariant: The Leader is always a member of the ISR.
  • Leader Election: Only replicas in the active ISR are eligible to become the new Leader during failover (when unclean.leader.election.enable = false).
  • Data Protection Standard: replication.factor = 3, min.insync.replicas = 2, acks = all.
  • Read Invariant: When writes are rejected due to $ \text{ISR} < \text{min.insync.replicas}$, consumer reads continue up to High Watermark without error.

Critical ISR Monitoring Metrics

| JMX Metric | Alert Severity | Production Meaning | | :— | :— | :— | | UnderReplicatedPartitions | 🔴 Critical | Partitions where $|\text{ISR}| < \text{ReplicationFactor}$; replica is lagging or node is down. | | IsrShrinksPerSec | 🟡 Warning | Frequency of replicas being dropped from ISR (investigate GC, network, or disk I/O). | | IsrExpandsPerSec | 🟡 Warning | Frequency of replicas catching up and returning to ISR (high values indicate flapping). | | UnderMinIsrPartitionCount | 🔴 Critical | Partitions where $|\text{ISR}| < \text{min.insync.replicas}$; write operations are actively blocked! |

Red Flags to Avoid

  • ❌ “The ISR consists only of follower replicas, while the leader is managed separately.” (The Leader is always part of the ISR).
  • ❌ “If ISR drops below min.insync.replicas, both reads and writes are blocked.” (Reads continue unrestricted up to the High Watermark).
  • ❌ “A follower is dropped from ISR if it falls behind by more than 10,000 records.” (Message count thresholds were abolished; ISR is evaluated purely on elapsed time via replica.lag.time.max.ms).